System and method for processing human related data including physiological signals to make context aware decisions with distributed machine learning at edge and cloud

ABSTRACT

A system and method for processing human related data to make personalized and context aware decisions with distributed machine learning at an edge and a cloud is disclosed. A nearest edge computing device receives first, second and third sensed signals from first, second and third sensory devices, determines when the first, second and third sensed signals exceed corresponding thresholds, correlates pairs of the sensed signals to generate multiple correlation patterns, determines a lag time between the first sensed signal exceeding the first threshold and the second sensed signal exceeding the second threshold, provides each of the multiple correlation patterns and the lag time as inputs to multiple long short term memory (LSTM) neural networks, controls the multiple LSTM neural networks to provide outputs, and maps the patient to a stage of a medical condition based at least in part on the multiple correlation patterns and the lag time.

INCORPORATION BY REFERENCE TO ANY PRIORITY APPLICATIONS

Any and all applications for which a foreign or domestic priority claimis identified in the Application Data Sheet as filed with the presentapplication are hereby incorporated by reference under 37 CFR 1.57.

This application claims benefit of U.S. Provisional Application No.62/926,335, filed on Oct. 25, 2019, entitled System and Method forProcessing Human Related Data Including Physiological Signals to MakeContext Aware Decisions with Distributed Machine Learning at Edge andCloud.

BACKGROUND Field

The described technology generally relates to artificial intelligence inmedical decision making, and in particular processing humanphysiological signals over varying periods of time to make a medicaldecision.

Description of the Related Technology

The use of computerized medical decision support in a hospital orclinical setting is known.

SUMMARY

One inventive aspect provides prediction of acute heart failure orprediction of other adverse events in heart diseases (e.g., orthostatichypertension, myocardial infarction).

Another aspect is automated diagnosis of four different classes of heartfailure and classification of all heart failure patients into these fourclasses based on guidelines of heart health related organizations suchas the New York Heart Association or the American College of Cardiology.

Another aspect is to classify all heart failure patients into aparticular subtype, such as reduced ejection fraction (EF) or preservedEF; systolic vs. diastolic heart failure; right ventricular vs. leftventricular heart failure (such as using a semi-supervised or scatteringembodiment).

Another aspect is to provide edge technology for advanced telemedicinefor a cardiologist and internal medicine (primary care). This technologyenables healthcare professionals to see all physiological signals in ahistory between visits, to store notes and data from each physicalexamination, and aggregate all the information in the edge for anomalydetection of blood pressure (BP), heart rate (HR), heart ratevariability (HRV). This can be used for prevention as well as fordiagnosis or prediction of adverse events.

Physiological signal changes may happen over an extended period of time(an hour or more) at least a few hours prior to an acute heart failure.The described technology can detect the changes in real time and let thedoctor intervene in a timely manner to avoid adverse events such asacute heart failure.

In some embodiments, the described technology's correlation, multi-levelrecurrent neural network (RNN) and long short-term memory (LSTM) with anattention network and a memory aggregator can learn and detect a longtemporal history of physiological signal changes in a novel way andtherefore identify the patient's risk.

For example, correlation of signals and multi-level LSTM can learnlonger term temporal history than any other method by utilizing theattention network and the memory aggregator. A two dimensional (2-D)attention heat map learned over multiple signals and multiple states canprovide an interpretable artificial intelligence (AI) result that canexplain which portion of input signals or features result in thedecision that the AI system makes. The interpretable AI can readily beexplained to a doctor and builds their trust for adoption better thanany other black box AI. Due to the method using correlation betweensignals and the interpretable attention heat map, the system needs muchless data to train the network. A group of cardiologists can help tolabel recorded data and suggest new intermediate nodes to the machinelearning (ML) method to help explain decision making of the ML(AI)models. The system can include multiple processes for decision making.

Another aspect relates to a system for processing human related data tomake personalized and context aware decisions with distributed machinelearning at an edge and a cloud, the system comprising a plurality ofedge computing devices configured to communicate data with each other,the plurality of edge computing devices physically spaced apart fromeach other; at least three sensory devices comprising first, second andthird sensory devices configured to sense a patient's physiologicalsignal in real time to generate a first sensed signal, a second sensedsignal and a third sensed signal and communicate the first, second andthird sensed signals to an edge computing device nearest available tothe first, second and third sensory devices among the plurality of edgecomputing devices; and a core cloud network configured to communicatewith the edge computing devices or the at least three sensory devices,the nearest available edge computing device being in data communicationwith the core cloud network and configured to receive the first, secondand third sensed signals from the first, second and third sensorydevices; determine when the first sensed signal exceeds a firstthreshold for a first predetermined time; determine when the secondsensed signal exceeds a second threshold for a second predeterminedtime; determine when the third sensed signal exceeds a third thresholdfor a third predetermined time; correlate the first sensed signal andthe second sensed signal to generate a first correlation pattern;determine a lag time between the first sensed signal exceeding the firstthreshold and the second sensed signal exceeding the second threshold;provide the first correlation pattern and the lag time as inputs to afirst long short term memory (LSTM) neural network; correlate the secondsensed signal and the third sensed signal to generate a secondcorrelation pattern; provide the second correlation pattern to a secondLSTM neural network as an input; control the first LSTM neural networkand the second LSTM neural network to provide outputs; and map thepatient to a stage of a medical condition based at least in part on thefirst correlation pattern, the lag time and the second correlationpattern.

The nearest available edge computing device may be further configured tocorrelate the first sensed signal and the third sensed signal togenerate a third correlation pattern; provide the third correlationpattern to a third LSTM neural network as an input; collect a history ofstates from each of the first, second and third LSTM neural networks;analyze the history of the states using an attention network such thatan output of the attention network learns interactions across time andacross signals; and summarize a history of the interactions using amulti-signal memory aggregator such that an output of the multi-signalmemory aggregator is fed into a decision making module to map thepatient to the stage of the medical condition based on the summarizedhistory of the interactions.

The nearest available edge computing device nay comprise a first featureextractor configured to determine when the first sensed signal exceedsthe first threshold for the first predetermined time; a second featureextractor configured to determine when the second sensed signal exceedsthe second threshold for the second predetermined time; a third featureextractor configured to determine when the third sensed signal exceedsthe third threshold for the third predetermined time; and a firstcorrelator configured to correlate the first sensed signal and thesecond sensed signal to generate the first correlation pattern anddetermine the lag time between the first sensed signal exceeding thefirst threshold and the second sensed signal exceeding the secondthreshold as inputs to a first cell in the first LSTM neural network inan LSTM bank, wherein the third feature extractor is configured todirectly feed the third sensed signal to a first cell in the second LSTMneural network in the LSTM bank, wherein the first correlator is furtherconfigured to generate additional first correlation patterns over timeinto additional different cells of the first LSTM neural network,wherein the third feature extractor is further configured to provideadditional instances over time when the third sensed signal exceeds thethird threshold as input signals into additional different cells of thesecond LSTM neural network, wherein the cells of each of the first LSTMneural network and the cells of the second LSTM neural network in theLSTM bank are configured to be fed into a fully connected neural networkto generate attention map coefficients that are component-wisemultiplied with the cells of the first LSTM neural network and the cellsof the second LSTM neural network to generate an attention map, whereinthe attention map is configured to be fed into the multi-signal memoryaggregator to aggregate multiple signal memories over time to prepare anoptimal input into the decision making module, and wherein the decisionmaking module is configured to make a decision to map the patient to thestage of the medical condition based on the optimal input received fromthe multi-signal memory aggregator.

The first sensed signal, the second sensed signal and the third sensedsignal may be of different modalities, wherein the first correlationpattern, the second correlation pattern, a state of the first LSTMneural network and a state of the second LSTM neural network may beconfigured to be fed into a first multi-modal LSTM neural network,wherein the second correlation pattern, the third correlation pattern,the state of the second LSTM neural network and a state of the thirdLSTM neural network may be configured to be fed into a secondmulti-modal LSTM neural network, and wherein the states of the first,second and third LSTM neural networks, and outputs of the firstmulti-modal LSTM neural network and the second multi-modal LSTM neuralnetwork may be configured to be fed into a multi-signal memoryaggregator.

The system may further comprise an attention function processorconfigured to receive one of the first, second and third sensed signalsas an input signal; find one or more certain patterns of the inputsignal; and categorize the input signal and generate the attention mapcorresponding to the certain patterns before being correlated.

The system may further comprise an attention function processorconfigured to receive one of the first, second and third sensed signalsas an input signal; find one or more certain patterns of the inputsignal; and categorize the input signal and generate the attention mapcorresponding to the certain patterns before being an input of one ofthe first or second multi-modal LSTM neural networks.

The nearest available edge computing device may be configured to presentthe attention map to a healthcare professional as documentation tosupport the determination of the stage of the patient's medicalcondition.

The decision making module may comprise at least one fully connectedneural network. The decision making module may be configured to generatea scalar quantified risk score. The fully connected neural network maycomprise a scaled sigmoid activation function. The decision makingmodule may comprise an argmax function configured to operate on anoutput of the fully connected neural network. The decision making modulemay be configured to generate a binary format prediction. The fullyconnected neural network may comprise a unit for each class of amultiple-class classification and wherein the output of the argmaxfunction is a probability of the input data belonging to each class ofthe multiple-class classification.

The nearest available edge computing device may comprise at least one ofthe first to third LSTM neural networks and the decision making module.The nearest available edge computing device may be configured to bufferand align at least one of the first, second and third sensed signalsbefore being correlated.

In another aspect there is an edge computing device for processing humanrelated data to make personalized and context aware decisions withdistributed machine learning at an edge and a cloud, the edge computingdevice comprising a memory storing computer executable instructions; anda processor in data communication with the memory and, when executed bythe executable instructions, configured to receive a first sensedsignal, a second sensed signal and a third sensed signal obtained inreal time from sensing a patient's physiological signal from first,second and third sensory devices, determine when the first sensed signalexceeds a first threshold for a first predetermined time, determine whenthe second sensed signal exceeds a second threshold for a secondpredetermined time, determine when the third sensed signal exceeds athird threshold for a third predetermined time, correlate the firstsensed signal and the second sensed signal to generate a firstcorrelation pattern, determine a lag time between the first sensedsignal exceeding the first threshold and the second sensed signalexceeding the second threshold, and correlate the second sensed signaland the third sensed signal to generate a second correlation pattern; afirst long short term memory (LSTM) neural network configured to receivethe first correlation pattern and the lag time from the processor; and asecond long short term memory (LSTM) neural network configured toreceive the second correlation pattern from the processor, wherein theprocessor is further configured to control the first LSTM neural networkand the second LSTM neural network to provide outputs; and map thepatient to a stage of a medical condition based at least on the firstcorrelation pattern, the lag time and the second correlation pattern.

The processor may be further configured to correlate the first sensedsignal and the third sensed signal to generate a third correlationpattern; and provide the third correlation pattern to a third LSTMneural network as an input, wherein the processor may be configured tomake a decision on outputs of the first, second and third LSTM neuralnetworks. The processor may be further configured to make the decisionby performing a scattering function on the outputs of the first, secondand third LSTM neural networks.

The processor may be further configured to correlate the first sensedsignal and the third sensed signal to generate a third correlationpattern; provide the third correlation pattern to a third LSTM neuralnetwork as an input; collect a history of states from each of the first,second and third LSTM neural networks; analyze the history of the statesusing an attention network such that an output of the attention networklearns interactions across time and across signals; summarize a historyof the interactions using a multi-signal memory aggregator; and feed anoutput of the multi-signal memory aggregator into a decision makingmodule to map the patient to the stage of the medical condition based onthe summarized history of the interactions.

The processor may comprise a first feature extractor configured todetermine when the first sensed signal exceeds the first threshold forthe first predetermined time; a second feature extractor configured todetermine when the second sensed signal exceeds the second threshold forthe second predetermined time; a third feature extractor configured todetermine when the third sensed signal exceeds a third threshold for athird predetermined time; and a first correlator configured to correlatethe first sensed signal and the second sensed signal to generate thefirst correlation pattern and determine the lag time between the firstsensed signal exceeding the first threshold and the second sensed signalexceeding the second threshold as inputs to a first cell in the firstLSTM neural network in an LSTM bank, wherein the third feature extractoris configured to directly feed the third sensed signal to a first cellin the second LSTM neural network in the LSTM bank, wherein the firstcorrelator is further configured to generate additional firstcorrelation patterns over time into additional different cells of thefirst LSTM neural network, wherein the third feature extractor isfurther configured to provide additional instances over time when thethird sensed signal exceeds the third threshold as input signals intoadditional different cells of the second LSTM neural network, whereinthe cells of each of the first LSTM neural network and the cells of thesecond LSTM neural network in the LSTM bank are configured to be fedinto a fully connected neural network to generate attention mapcoefficients that are component-wise multiplied with the cells of thefirst LSTM neural network and the cells of the second LSTM neuralnetwork to generate an attention map, wherein the attention map isconfigured to be fed into a multi-signal memory aggregator to aggregatemultiple signal memories over time to prepare an optimal input into adecision making module, and wherein the decision making module isconfigured to make a decision to map the patient to the stage of themedical condition based on the optimal input received from themulti-signal memory aggregator.

In yet another aspect, there is a method of processing human relateddata to make personalized and context aware decisions with distributedmachine learning at an edge computing device in communication with acloud, the method comprising receiving, at a processor of the edgecomputing device, a first sensed signal, a second sensed signal and athird sensed signal obtained from sensing a patient's physiologicalsignal from first, second and third sensory devices; determining, at theprocessor, when the first sensed signal exceeds a first threshold for afirst predetermined time, determining, at the processor, when the secondsensed signal exceeds a second threshold for a second predeterminedtime, determining, at the processor, when the third sensed signalexceeds a third threshold for a third predetermined time; correlating,at the processor, the first sensed signal and the second sensed signalto generate a first correlation pattern, determining, at the processor,a lag time between the first sensed signal exceeding the first thresholdand the second sensed signal exceeding the second threshold;correlating, at the processor, the second sensed signal and the thirdsensed signal to generate a second correlation pattern; receiving, at afirst long short term memory (LSTM) neural network of the edge computingdevice, the first correlation pattern and the lag time; receiving, at asecond LSTM neural network of the edge computing device, the secondcorrelation pattern; controlling, at the processor, the first LSTMneural network and the second LSTM neural network to provide outputs;and mapping, at the processor, the patient to a stage of a medicalcondition based at least on the first correlation pattern, the lag timeand the second correlation pattern.

The method may further comprise correlating, at the processor, the firstsensed signal and the third sensed signal to generate a thirdcorrelation pattern; receiving, at a third LSTM neural network of theedge computing device, the third correlation pattern; collecting, by theprocessor, a history of states from each of the first, second and thirdLSTM neural networks; analyzing, at an attention network of the edgecomputing device, the history of the states to learn interactions acrosstime and across signals; summarizing, at a multi-signal memoryaggregator of the edge computing device, a history of the interactions;feeding, by the processor, an output of the multi-signal memoryaggregator into a decision making module of the edge computing device;and mapping, at the decision making module, the patient to the stage ofthe medical condition based on the summarized history of theinteractions.

The method may further comprise receiving first correlation patterninputs at a first cell of the first LSTM neural network in an LSTM bank;directly receiving the third sensed signal when the third sensed signalexceeds the third threshold at a first cell in the second LSTM neuralnetwork in the LSTM bank; receiving additional first correlationpatterns over time into additional different cells of the first LSTMneural network; receiving, by the processor, additional instances overtime when the third sensed signal exceeds the third threshold as inputsignals into additional different cells of the second LSTM neuralnetwork; generating, by an attention network of the edge computingdevice, attention map coefficients based on the cells of each of thefirst LSTM neural network and the cells of the second LSTM neuralnetwork in the LSTM bank to be fed into a fully connected neuralnetwork; generating, by the attention network, an attention map based onthe attention map coefficients that are component-wise multiplied withthe cells of the first LSTM neural network and the cells of the secondLSTM neural network; feeding, by the processor, the attention map into amulti-signal memory aggregator that is configured to aggregate multiplesignal memories over time; and mapping, at a decision making module, thepatient to the stage of the medical condition based on the aggregatedmultiple signal memories received from the multi-signal memoryaggregator.

The edge computing device may be a nearest available edge computingdevice to the patient among a plurality of edge computing devicescomprising the first to third edge computing devices being in datacommunication with the cloud.

The method may further comprise receiving a request for service from asensory device of the patient; locating the patient's sensory device;determining that the edge computing device is a nearest available edgecomputing device to the patient sensory device of a plurality of edgecomputing devices comprising the first to third edge computing devicesbeing in communication with the cloud; and assigning a service slot tothe patient's sensory device. Determining the nearest available edgecomputing device may comprise receiving, at the plurality of edgecomputing devices, a signal sent by the patient's sensory device;measuring strengths of the signal received by the plurality of edgecomputing devices; comparing the strengths of the received signal; anddetermining an edge computing device to have a strongest signal strengthas the nearest available edge computing device.

The method may further comprise buffering and aligning one of the firstsensed signal or the second sensed signal before the correlating. Themethod may further comprising finding one or more certain patterns ofthe first sensed signal or the second sensed signal; and generating anattention map corresponding to the certain patterns before thecorrelating. The method may further comprise presenting the attentionmap to a healthcare professional as documentation to support thedetermination of the stage of the patient's medical condition.

In another aspect, there is a system for processing human related datato make personalized and context aware decisions with distributedmachine learning at an edge and a cloud, the system comprising aplurality of edge computing devices configured to communicate data witheach other, the plurality of edge computing devices physically spacedapart from each other; at least two sensory devices comprising first andsecond sensory devices configured to sense a patient's physiologicalsignal in real time to generate a first sensed signal and a secondsensed signal and communicate the first and second sensed signals to afirst edge computing device nearest available to the first and secondsensory devices among the plurality of edge computing devices; and acore cloud network configured to communicate with the edge computingdevices or the at least two sensory devices, the nearest edge computingdevice in data communication with the core cloud network and configuredto receive the first and second sensed signals from the first and secondsensory devices; determine when the first signal exceeds a firstthreshold for a first predetermined time and subsequently determine whenthe second signal exceeds a second threshold for a second predeterminedtime; correlate the first signal and the second signal to generate afirst correlation pattern; determine a lag time between the first signalexceeding the first threshold and the second signal exceeding the secondthreshold; and provide the first correlation pattern and the lag time asinputs to at least two recurrent neural networks (RNNs) operativelyconnected to each other to provide an input to a decision making moduleto map the patient to a stage of a medical condition based at least onone or more of the first correlation patterns and the lag time.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high level block diagram of an embodiment of a system fordecision making using distributed edge computing and machine learning.

FIG. 2 is a block diagram of an embodiment for processing humanphysiological signals such as performed on the system of FIG. 1.

FIG. 3 is a block diagram of an embodiment for processing humanphysiological signals through decision making such as performed on thesystem of FIG. 1.

FIG. 4A is a block diagram of an embodiment of an edge machine learningsystem such as identified in FIG. 1.

FIG. 4B is a block diagram of an embodiment of a core network/cloudhealth analytic services such as identified in FIG. 1.

FIG. 4C is a high level block diagram of another embodiment of a systemfor decision making using distributed edge computing and machinelearning.

FIG. 5 is a block diagram of another embodiment for processing humanphysiological signals through decision making such as performed on thesystem of FIG. 1.

FIG. 6 is a block diagram of another embodiment for processing humanphysiological signals through decision making such as performed on thesystem of FIG. 1.

FIG. 7 is a block diagram of another embodiment for processing humanphysiological signals through decision making such as performed on thesystem of FIG. 1.

FIG. 8 is a block diagram of another embodiment for processing humanphysiological signals through decision making such as performed on thesystem of FIG. 1.

FIG. 9 is a block diagram of another embodiment for processing humanphysiological signals through decision making such as performed on thesystem of FIG. 1.

FIG. 10 is a block diagram of an embodiment for processing humanphysiological signals through decision making and including an exampleattention network and an example multi-signal memory aggregator such asperformed on the system of FIG. 1.

FIG. 11 is a block diagram of another embodiment for processing humanphysiological signals through decision making and including theattention network and the multi-signal memory aggregator such asperformed on the system of FIG. 1.

FIG. 12 is a block diagram of another embodiment for processing humanphysiological signals through decision making and including theattention network and the multi-signal memory aggregator such asperformed on the system of FIG. 1.

FIG. 13 is a block diagram of another embodiment for processing humanphysiological signals through decision making and including theattention network and the multi-signal memory aggregator such asperformed on the system of FIG. 1.

FIG. 14 is a block diagram of another embodiment for processing humanphysiological signals through decision making and including theattention network and the multi-signal memory aggregator such asperformed on the system of FIG. 1.

FIG. 15 is a block diagram of the example attention network andmulti-signal memory aggregation in more detail.

FIG. 16 is a block diagram of an example attention map at a particulartime step and multi-signal memory aggregation in more detail.

FIG. 17A is a block diagram of an embodiment of decision making thatoutputs a scalar.

FIG. 17B is a block diagram of an embodiment of decision making thatoutputs a binary decision.

FIG. 17C is a block diagram of an embodiment of decision making thatoutputs a multi-class decision.

FIG. 18 is a block diagram illustrating an embodiment of generating anattention map using a fully connected neural network.

FIG. 19 is a flowchart of an embodiment of an example flow for buildingan initial model architecture for a particular objective.

FIG. 20 is a flowchart of an embodiment of an example flow forprocessing two human physiological signals.

FIG. 21 is a block diagram of an embodiment for a portion of an examplemulti-modal (MM) long short-term memory (LSTM) showing the fusion ofLSTM states as input, and signals as separate input.

FIG. 22 is a block diagram of an embodiment for a portion of an exampleMM-LSTM showing how both signals and hidden states from other LSTMs canbe combined inside of the MM-LSTM.

FIG. 23 is a block diagram of an embodiment for a portion of an examplemulti-level modified (MLM) recurrent neural network (RNN) illustratingan example way to combine information including hidden states betweenseparate RNNs.

DETAILED DESCRIPTION OF CERTAIN EMBODIMENTS

As described in various example embodiments, a system and method forprocessing human related data including physiological signals aredisclosed to make context aware decisions with distributed machinelearning at edge and cloud. Although the example embodiments aredescribed with respect to a particular system for decision making usingdistributed cloud and edge computing and machine learning, the describedtechnology is not limited to the disclosed embodiments.

FIG. 1 is a high level block diagram of an embodiment of a system 100for decision making using distributed cloud and edge computing andmachine learning. The decision making system 100 shown in FIG. 1 ismerely an example, and can have different structures, shapes, and/oruser interfaces. The components of the system 100 may be directly orindirectly connected to each other. The components of the system 100 mayalso be in wired or wireless connection with each other. Furthermore,certain components may be removed (e.g., optional) or others can beadded to the decision making system 100, and this can be applied to theblock diagrams of the other figures. The system 100 includes a corenetwork/cloud health analytic services system 110, a privatecloud/hospital server 120, an edge machine learning (ML) system 130 andone or more other edge ML systems 150, and an edge ML system 160.

In some embodiments, the edge ML system 130 and the one or more otheredge ML systems 150 interconnect with the core network cloud healthanalytic services system 110 by wired or wireless connections. Wiredconnections may include use of a local area network, wide area network,the Internet and others, and may include use of the Ethernet or otherstandards. Wireless connectivity may include use of Wi-Fi or cellularconnections using 4G, LTE, 5G or other standards.

The system 100 can also include one or more fixed or mobile devices,such as a camera or video camera 132, and devices to measure certainhuman physiological signals, including, but not limited to, anelectroencephalogram (EEG) 142, an electrocardiogram (ECG) 144,respiration 146 and blood pressure 148 for an indoor patient 140 or anoutdoor patient 152. In some embodiments, these devices 132 and 142-149communicate data with one or more edge devices, such as 130 or 150, inits vicinity using, for example, wireless or wireline protocols.

The edge ML system 130 can be located in a patient's home. It canreceive physiological signals captured by sensor devices such aswearables or patches located on a patient body or implants inside theirbody through one or more wireless protocols. These sensor devices can beinitialized or configured by over the air software update.

The system 100 can also include a display 134 connected to the edge MLsystem 130 with wired, wireless or wireline protocols. Captured signalsmay include, but are not limited to, ECG, photoplethysmography (PPG),respiration, bioimpedance of a lung or other part of body (congestion),blood pressure (BP), pulse oximeter (SPO₂ blood oxygen level),electromyography (EMG), EEG, physical activity or accelerometer data,face expression, angle of arrival and or time of arrival to locatepatient and/or depth information, heart beating, and voice or any audiosignal from patient including background noise. The outdoor patient 152wearables or patches may communicate with the edge ML system 150 or anynearer edge ML system directly or through a phone or watch.

The sensor device periodically sends a unique signal such as a beaconand all edge systems in a vicinity receive that signal and measure thesignal strength of the received signal. The edge systems coordinate witheach other the measured signal strengths and the edge system with ahighest received signal strength designates a channel for that sensorydevice to start a link and communicate with the edge system.

The private cloud/hospital server 120 can be in wired or wirelesscommunication with the edge system 160 that may include machine learningand augmented reality (AR)/virtual reality (VR). In institutions such asa hospital or outpatient clinics (doctor offices) one can use the edgesystem 160, for example, to have AR/VR capability for performing remoteprocedures with healthcare professionals including but not limited todoctors, physician assistants or nurses for advance telemedicine orcolonoscopy or other services.

In some embodiments, as part of the core network/cloud health analyticservices system 110 (hereinafter to be interchangeably used with thecloud), a cloud monitoring center providing health and analyticsservices receives all physiological data from a large number of edgedevices located in patients' homes and stores data in its database (seeFIG. 4B). The cloud 110 can be platform independent (using container, itcan be run on Amazon AWS, Microsoft Azure, Google, or other providers)and through the core network it may connect to the privatecloud/hospital servers 120 of other institutions including, but notlimited to, health systems, hospitals, outpatient clinics, medicalgroups, transitional care, nursing homes, rehabilitation centers, andhome health care agencies. The cloud monitoring center may integrateelectronic medical records (EMR) of any institution such as healthsystems or hospitals through a flexible API (such as shown in FIG. 4B).

Each mobile device can go through a discovery mode when it is turned onor when it wakes up from sleep by sending a request for service. Edgedevices, such as 130 and 150, may be in listening mode and afterlocating the new device, the closest edge may assign a service slot tothe new device. Each edge device can be connected to a core network(cloud) 110 through a communication link. In addition, adjacent edgedevices can communicate directly for lower latency applications when amobile device needs to be handed off to the new edge device and it istraveling fast, for example. These fixed or mobile devices could be anydevice including, but not limited to, wearables (such as watches,garment, belts or other wearable devices), patches or sticks on thebody, implants inside body, phone, video camera, sensors (temperature,air pressure, air quality), actuators, robots, tablet, laptop, TV,display, appliances, drones, cars, buses (and cameras on them), trains,bikes, scooters, and motor cycles.

The edge ML systems 130 and 150 can be a gateway or hub that has machinelearning and decision-making capability and can provide differentautomated services. In one embodiment, referred to as health management,it can merge all data from cameras and other sensing devices such asvoice and vital signs and perform semi-supervised learning algorithm todetermine face expression, emotion, the health and safety of patient orsuggest a right diet. In another embodiment, at least one of the edge MLsystems 130 and 150 can make decisions with minimum latency for seriousproblems such as prediction of acute heart failure and provide insightwith an interpretive report to the doctor to intervene and takecorrective action. At least one of the edge ML systems 130 and 150 canincorporate any model and parameters from supervised learning of abigger data set residing in the cloud 110. The edge gateway may belocated inside home, at light pole in street, in the car or in thehospital and can be connected to the core network through wired orwireless communication and can be updated over the air. The latest modeland parameters can be pushed to the edge ML systems 130 and 150 over theair that provides service such as feedforward decision making regardingthe risk of acute heart failure (providing a risk score) or predictingacute heart failure. The latest AI or ML models and parameters can bepushed to the edge ML systems 130 and 150 that provide service remotelyinside a patient's home or inside a business, office or factory, forexample.

FIG. 4A is a block diagram of an embodiment of an edge machine learningsystem such as identified in FIG. 1. Example internal blocks of the edgeML systems 130 and 150 are shown in FIG. 4A. A quad-core ARM processor410 (four A53) is an example of a CPU inside the edge system 130, 150. Aneural processing unit (NPU) 420 is a hardware accelerator to perform,for example, Tensor operations required for neural network models. TheNPU 420 can be obtained from ARM or Samsung, for example. The NPU 420can perform the forward path (inference) of neural network models oninputs from multiple sensors at a much higher speed and can achievelower inference time compared to running on an ARM even with an existingsoftware accelerator such as an ARM NN. Therefore, the novel machinelearning algorithms described herein can operate on the edge in realtime and achieve very low latency. The edge system 130, 150 includes asufficient memory 430 to buffer sensory device inputs and enough RAM 432to process them in real time with the NPU 420 and CPU 410. The edgesystem 130, 150 may have a built-in 4G/5G/LTE-M cellular modem 450 thatcan receive over-the-air updates of models and transmit over-the-airmachine learning results. The edge system 130, 150 includes a built-inWi-Fi and BLE modem 452 for wireless communication with cameras or withwearables, patches and implants that capture physiological signals.

The edge system 130, 150 may also include multimedia interfaces such asan audio interface 440, a camera interface 442, a video encoder anddecoder 444, and a 3-D graphic accelerator 446. The edge system 130, 150may further include an SD card interface 434 (for flash memory) that canbe used to boot the edge, load applications or save data in case ofnetwork or Internet disconnection for later recovery.

State of the art machine learning models can be trained to run on theedge system 130, 150 for face expression, face recognition, imagesegmentation and processing to see inside a mouth or an ear of a patientfor advanced telemedicine.

The edge gateway can provide different services and connect to differentdevices through different radio access protocols depending on data rate,required mobility and latency required for that service. The edgegateway can be used for all three type of use cases in 5G: highthroughput use case, low latency use case, and large number of devicesuse case. Some applications or services may require a combination of twoor three 5G capabilities or use cases such as traffic control,industrial Internet of Things (IOT), remote surgery, smart home andhealth management or smart city.

To help with capacity and latency, different devices may be assigned todifferent beams depending to their location, where the beam directionand beam width are adaptive and can depend on locations of mobiledevices at a given time. On the edge system, a bigger antenna array maybe used rather than on the mobile device.

FIG. 4B is a block diagram of an embodiment of a core network/cloudhealth analytic services such as identified in FIG. 1. An overview of acloud architecture for the cloud 110 is illustrated in FIG. 4B. An AWSIoT Core 410 includes a group 412, a device, a core, a shadow 414, asubject 416 and a Lambda 418. Different edge devices can be assignedinto different groups 412 based on their intended use (e.g., heartfailure prediction). The shadow 414 of each edge device can bemaintained in the cloud, which makes the state of the device availableeven if the device is disconnected from the AWS IoT Core 410. Thisallows the system to continue to collect and report data about devicestate if devices go offline. It also allows other AWS services torequest changes to device state (like updating vital sign thresholds)even for offline devices; the state change requests are performed on thedevice shadow, and the device will sync to the shadow when it comes backonline.

The edge devices may communicate directly with the AWS IoT Core 410 viaMessage Queue Telemetry Transport (MQTT) messages. Each type of datamessage may be published to a separate subject 416. One or moresubscribers in the cloud can subscribe to each subject. This allows thesystem to evoke different Lambda functions 418 based on data type.Lambda functions can be automatically triggered to execute specific codeby predetermined events like vital sign threshold crossings, changes inpatient status, and data transmission. This allows the system toautomate many functions, including sending alerts, updating databases,and sending reminders. AWS S3 420 is used to trigger the AWS Lambda 418to immediately process incoming data after it is received over MQTTmessages. There are separate Lambda functions which push data to aPostgres database (DB) 456, an object-relational database system. ThePostgres DB 456 can be used to safely store and scale system uploadeddatasets. Built on PostgreSQL, the DB was selected based on its strongreputation for reliability, data integrity, and fault tolerance. WithPostgreSQL new data types (e.g. structured data types and documents) canbe created and custom functions (e.g. query planning and optimization)can be built for the system 100. The Postgres DB 456 can serve as asystem primary data store for system 100 web applications, patienttime-series data, and machine learning models.

The Postgres DB 456 can be connected with an EC2 server 450, whichprovides secure and resizable cloud computing, as well as hosting asystem Flask-based web app on a webserver 452. Each EC2 instance canperform all the functionalities of a traditional web server, with theadded benefit of having flexibility to provision servers on demand basedon the system's current computational requirements. The EC2 server 450can also run machine learning 458 on data stored in these databases. Thesystem's machine learning block 458 allows the core network 110 to embedmachine learning processing directly into system 100 SQL queries ascalls to functions. This also allows for training and deploying system100 models faster by leveraging the compute power of ML-optimized cloudservers. In addition to developer generated patient-oriented models likeheart failure prediction and classification, the block 458 can accessoff-the-shelf machine learning algorithms and services from AWS. Anexample is Amazon SageMaker, which can help to automate the explorationof new and improved models by using its built-in tools whichautomatically build, train, and tune machine learning models.

The system web app queries the Postgres DB 456 when populatinginformation on the various pages. To integrate with various EMR andother health systems, the system uses their 3rd party APIs 440 to sendand receive data between the system platform via an API engine 454connected to the Postgres DB 456 and the platforms of the EMR and otherhealth systems. This allows the system 100 to directly and securelyinherit patient's medical records.

The specific API is dependent upon the particular third party, but eachAPI allows for directly and securely inheriting patient's medicalrecords. The patient's records may be parsed to initialize certain riskmodels for each patient, including open-source random forests trained toclassify patients into risk category based on information in theirmedical records including age, sex, and history of smoking. Systeminternal APIs are maintained and secured in the system API engine 454,which can perform functions such as calling SQL queries and interactingwith third party APIs 440. Examples of system 100 APIs include messagebrokers that provide interoperability between the system webapplication, internal databases, and 3rd party APIs.

The edge devices 130, 150 can provide health management from preventionand early diagnosis to chronic disease management to saving life bymaking an action in real time by integration and (decision making)perception capability. The edge enables and integrates this healthmanagement service as part of daily life, and this service can lowerrising health care costs. Senior people can enjoy their life in theirhome using this technology, and also patients discharged from a hospitaldo not need to get readmitted every few days.

The edge device allows for monitoring an individual and environmentaround him/her, analyzing a state of his/her health and adjustingmedication, diet and entertainment to give the individual comfort withminimum effort from him/her or their family. The edge technology enablesindependent living in their home or in their suite as part of a seniorcommunity and reduces costs for family and Medicare.

Embodiments of the described technology describe how edge computing isused in the system 100, such as by applying artificial intelligence (AI)and machine learning at the edge to make the health management doableand cost effective at home and even in a car and around a city.

FIG. 4C is a high level block diagram of another embodiment of a systemfor decision making using distributed edge computing and machinelearning. For remote procedures, small surgery or even advancedtelemedicine an edge to edge connection as shown in FIG. 4C takingadvantage of a 5G core network and open radio access network (0-RAN) toachieve ultra low latency end-to-end and reliability that no otherremote patient monitoring with AI solution currently offers. A physicianand patient may feel they are present in a same room by the doctorlooking at his/her mouth or ear via a camera 132 and listen to his heart143 with interfaces that the edge provide as seen in FIG. 4C. A patientcan leave messages for a nurse or doctor using the system speechrecognition or care giver can control time of SPO₂ measurement using apersonalized high accuracy speech recognition ML on the edge. In oneembodiment, the system 100 can provide an automated orthostatic BP testwhere the edge talks with the patient and measures BP at two differentpositions, sitting and standing, (by using ML on accelerometer data itcan recognize posture) and compare BP readings and send alarm if thedifference is outside threshold set by a doctor. In certain embodiments,an interactive audio capability allows the edge system 130, 150 todiscuss symptoms and feelings with patient using a speaker 145 and amicrophone 147. The edge may then cross check answers with physiologicalsignal measurements and detected events such as HRV<X or BP>Y or anykind of arrhythmia and make a comprehensive report to a doctor or nurseor care giver. Other interfaces include a BTLE interface with variouswearables 141, and an interface for robots or grips 151. The edge system160 can be connected to a VR console 162, a smartphone 160 and a tablet164 (e.g., iPad). An edge box in a user's home or car or on the streetpole can run one of the versions of the development's software algorithmto make sure of continuity of their health management service. Differentfeatures and configurations can be enabled and used in the version ofsoftware pushed from the core network 110 to an edge box at home versusan edge box at the street pole or in the car.

The edge can discover new paradigms in diagnosis and treatment follow-upusing unsupervised learning on personal physiological signals whiletaking advantage of learned baselines from a bigger population in whatis called evidence-based personalized medicine. The edge is a personalassistant to a patient and care giver including a doctor by bringing totheir attention the discovered results from analyzing signals over timeand letting the doctor make informed decisions, uncover the unknowns andthe right personalized treatment.

The edge can have a user's diet information every day, for example, in asmart home setting from a refrigerator (such as using weight sensorsand/or camera inside the refrigerator), physiological signals and theuser's voice and face expression. The edge can discover a correlationbetween the user's health, physiological signals such as ECG andhappiness with their diet, breathing, sleep and music listened to, suchthat the edge learns about the user and reinforces a good diet orfavorite music to get good sleep, health and a happy mood. If the user'sinput quality degrades one day, as detected by a change in quality ofdiet or breathing or sleep for example, and their health conditiondegrades following that (such as an ECG irregularity), then the edgelearns the weight of each input and can model their health condition,predict a future degradation of condition, determine the cause ofproblem and inform the user and their doctor to select the righttreatment.

Cross correlation of any two (or a greater number of) signals measuredover time such as heart rate, HRV (stress), blood pressure, oxygensaturation level, respiration rate, physical activity (type and stepcount), sleep quality and heart rhythm (ECG) irregularity (arrhythmiapercentage over time), and finding multiple unique correlation patternsthat can be shown to have been repeated in a person, and using thesefeatures to predict CHF (a composite risk score or binary prediction),and treating the issue before resulting in heart failure is desired.Discovering these physiological signal changes few hours beforeshortness of breath and other symptoms of HF happen that can indicate arisk factor for heart failure and reporting the risk level to aphysician can be done. Personalized medicine and evidence-baseddiagnosis can happen by using edge technology, thereby reducing risk andmistakes due to trial and error treatment and a lack of right diagnosis.

This system 100 and service can function as a health advisor to anyperson and as an assistant to a doctor. Because heart arrhythmias arecomplex and may have underlying or contributing causes related tolifestyle choices, the developer is uncovering these previously unknownunderlying or contributing causes by using a correlation neural networkscheme so as to help doctors to address their patients' health needs.

FIG. 2 is a block diagram of a system 200 for processing humanphysiological signals such as performed on the system of FIG. 1. In someembodiments, a data set in the cloud 110 is used to do supervisedmachine learning in the cloud and downloads parameters to the edgesystem 130, 150. Then, the described technology may continue with feedforward (FF) detection for face expression and unsupervised machinelearning and correlation discovery for evidence based and personalizedmedicine shown in FIG. 2.

Referring to FIG. 2, wireless modems 210, 220, 230, 240 receivedifferent signals 212, 222, 232, 242 such as ECG, BP, SPO2 andaccelerometers from sensory devices 144, 148 located on patient 140 asshown in FIG. 1. The system 200 may also include a video camera modem250 that receives a patient's face image and provides the face video tothe anomaly detection block 254, and a microphone 260 that provides apatient's voice to a voice recognition block 264. Preprocessing andanomaly detection blocks or modules 214, 224, 234, 244, 254 areperformed on these signals. As described in FIG. 9 and FIG. 10hereinbelow, at least one of the anomaly detection modules 214, 224,234, 244, and 254 may be realized with or include, for example,threshold detectors 920, 1020 or arrhythmia detectors 910, 1010 usingclassification on ECG or low activity detection using random forest andattention classification 922, 1022 on accelerometers data. The output ofthe anomaly detection modules 214, 224, 234, 244, 254 are fed to acorrelation network 270 to determine correlations between signalsincluding, but not limited to, HR, HRV, arrhythmia, BP, SPO₂, andactivity described in the description of FIG. 3. A multi-level modifiedRNN (MLM RNN) block 280 may include a bank of Recurrent Neural Networks(RNNs) that has interactions with each other based on two featuresdescribed in detail in conjunction with FIG. 3. The decision makingblock 290 has been described in detail at FIG. 3, and in one embodimentit could perform dynamic scattering as described at FIG. 3.

In some embodiments, the described technology covers algorithms andmethods to detect bio-signals dependencies as some bio-markers can beused for prevention, early diagnosis and treatment (precision medicine).

In some embodiments, the system 100 detects when a patient's vital signsdeteriorate (anomaly detection), for example, by using thresholddetectors, or a modified attention network, or arrhythmia detection asdescribed in conjunction with FIG. 9. The anomaly detection blocks (920,922) function as a switch that starts a sequence of events when theinput signal is determined to cross a certain threshold. Examples mayinclude, but are not limited to, resting heart rate above about 100 bpm,SpO₂ falling below about 85% or a decrease in SpO₂ by more than about 10points in a short time.

Referring to FIG. 3, inputs for anomaly detection blocks 300, 302, 304can include raw or processed data (including extracted features, such asheart rate) coming from sensor 1 to sensor x such as those describedwith respect to FIG. 1. Although FIG. 3 shows three anomaly detectionblocks and three sensors, in some embodiments, two sensors or more thanthree sensors can be used. When a bio-potential signal A (Sig_A) such asheart rate passes a certain (first) threshold (X_(A)) and stays there(high/low) for more than about a few minutes, it can be tagged as anevent, e.g., “activity A”. Then monitoring of other signals such asblood pressure, respiration rate, activity, facial expression and voicecan be performed to determine if any other abnormal activity followsactivity A and shows a consistent correlation.

When a bio-potential signal B (Sig_B) passes beyond a normal range or asecond threshold (or normal expression) (X_(B)), the correlation blocksor modules 310, 312, 314 can start correlation operations and measurecross correlation of stored activity with new activity (which can have alag) in real time. If a correlation value passes a certain threshold, anext neural network goes to a new state, increments a risk factor basedon a correlation peak between heart rate and systolic blood pressure,for example, and detects a time interval that this correlation valuestays up (active). The edge can record signal B activity and acorrelation of signal A and signal B activities. The system and methodcan measure a lag time interval between these two activities or anyother subsequent activities and can look for discovering a pattern thatrepeats itself for this individual.

In certain embodiments, every time both activity A and activity Bhappen, the correlation network can generate correlation values as afunction of time while two time series of signal A and signal B can bepresented as inputs to the network. As shown in FIG. 3, a correlationneural network is triggered to perform computation of these correlationsbetween every pair of inputs when abnormal events happen in both inputsto the correlation blocks 310, 312, 314, and it also computes lagbetween the two abnormal events based on a synchronized time stamp inthe edge system.

Correlation may be implemented by Equation 1:

C(n)=Σ_(m=n−w+1) ^(n) A _((m-k)) B _((m))  Equation 1

where

A=signal A,

B=signal B,

m=time index for summation over window of time w,

n=time index for output of correlation,

w=length of window to compute correlation, and

k=lag parameter between two signals.

Certain embodiments compute a correlation for any lag value when anomalydetection is not utilized.

The above equation is merely an example equation and other equations mayalso be used. The correlation blocks 310, 312, 314 can provide twouseful pieces of information: an amount of correlation between the twosignals as a function of lag time, and the lag associated with time ofmaximum correlation. The lag is represented by the time differencebetween signal A and signal B passing their respective thresholds.

Correlation can be computed over a time window w that can be dynamicallyset based on lag and anomaly detection thresholds and a length of timethat input A and B signals stay above the thresholds. In one embodiment,the w can be a hyperparameter that can be selected by training onoutputs of correlations for a given objective such as risk assessment ofacute heart failure. The time window can be the shortest of an activityA window and an activity B window (period that each signal/activitystays above threshold). Anomaly detection thresholds and correlationthresholds can be learned for a given disease or for a given individual.

FIG. 20 is a flowchart that depicts a process 2000 of a particularconfiguration of anomaly detection blocks. Although the process 2000 isdescribed herein with reference to a particular order, in variousembodiments, states herein may be performed in a different order, oromitted, and additional states may be added. This may apply to the otherflowcharts in the figures. These blocks are constantly processing theirinput signals to detect anomalies. In a step 2020, anomaly detection forsignal A 2010 or signal B 2015 begins. When a biopotential signal A 2010such as heart rate passes a certain threshold, and stays there(high/low) for more than about a few minutes, it can be tagged as ananomaly or event, e.g., “activity A”. This anomaly is detected at adecision step 2030. After that step, a buffer 2035 begins to fill withsamples of the signal in which the anomaly was detected. The othersignal, in this case signal B 2015, continues to be monitored foranomalies. If an anomaly is detected in this signal, at a decision step2040, it triggers a separate buffer 2045 to begin filling with samples.The lag “k” or, in other words, time difference between the anomalyevent detections is recorded at step 2050. Next, the cross-correlationof the A and B signals for lag “k” is computed at a step 2060 based on,for example, the equation 1 above. These correlation bio-markers arethen fed 2070 to the machine learning models described in differentembodiments herein.

A correlation network is a kind of dynamic feature computation from morethan one signal which triggers a next stage of system machine learningthat could be an interconnected multi-level modified recurrent neuralnetwork (RNN) 330, 332, 334.

Events that arrived at the edge system can be synchronized based onreceiving a time adjusted response from every device to a unique beacontransmitted from the edge.

RNNs are a class of neural networks specialized for processingsequential data, such as time-series. These networks can scale to longsequences, and can process sequences of variable length. RNNs can startwith some initialized state, and then operate by iterating over an inputsequence. At each time-step of the sequence, they combine the currentsequence element with the output from the previous time-step, andperform computation on this value to produce the next output.

The correlation networks 310, 312, 314 (see FIG. 3), plus a multi-levelmodified recurrent neural network (MLM-RNN) including an MLM-LSTM, candetermine whether correlation between activity A, activity B andactivity C are consistent, a same pattern repeats over time and the lagsbetween these physiological signal changes are unique for a givenperson.

There are many inputs or factors that can be narrowed down to a few mainfactors that cause a problem, such as a high oxygen demand vs oxygensupply, low activity, weak cardiac output and consequently acute heartfailure.

FIG. 23 is a block diagram of an embodiment for a portion of an examplemulti-level modified (MLM) recurrent neural network (RNN) illustratingan example way to combine information including hidden states betweenseparate RNNs. To find a relationship among correlations betweenactivities A, B, and C, the edge system can combine information from theseparate MLM-RNNs 320, 322, 324. This combination of information isrepresented by the vertical arrows between the MLM-RNNs (see also FIG.3). One way to combine information is to use a mutual exchange of hiddenstates between each RNN cell. In this way, at each time step, all cellscan receive the following as their input: the next output of thecross-correlation signal 320-1; their own previous output 320-2; and theoutput from the other RNNs (322-Ct, 324-Ct and so forth) as depicted inthe block diagram in FIG. 23. This diagram depicts a method forcombining information between separate RNNs. As a base case, two RNNsare illustrated.

For each RNN, its previous cell state can be routed through Wc, afully-connected neural network. Consider block 320 of FIG. 23 forexample. The next output of the cross-correlation signal is multipliedby Wy, and cell states from other RNNs are multiplied through Wa. Theoutputs of all three neural network gates are summed before being passedthrough a tan h function to yield the new cell state Ct. This method canbe extended to many RNNs. Each RNN can send its output Ct to all otherRNNs, as well as to itself. For receiving multiple inputs from otherRNNs, each RNN will concatenate these multiple Ct inputs into onevector, before passing it through its own gate Wa. For example, block320 receives multiple inputs at its gate Wa, as indicated by the text“322-Ct, 324-Ct, . . . ” on the arrow leading into Wa. This method ofcombining information can be used for combinations of signals withsimilar temporal dynamics. Since the cell states are shared between allRNNs, this method may provide the best performance when the inputsignals are varying along similar timescales.

A separate way to combine information from different RNNs is to have anobserver that does computation on a collection of states across theseparate RNNs. To achieve this, an N×T buffer of states can record thelast T states from each of the N RNNs. This tensor of states can beprocessed by a separate neural architecture, e.g., an attention moduleas in 1050 of FIG. 10. This strategy, depicted in FIG. 18, can be usedwhen there is a need to put attention on a long temporal history ofmultiple signals. In this method, an attention heatmap 1857 can begenerated in two dimensions, across states and across signal types.

Referring to FIG. 18, the cell states from the bank 1840 of RNNs arebuffered and passed into the attention network 1850, which findsinteractions across the different signals and across states; theseinteractions are reflected in each of the two dimensions of its output.To compute the attention coefficients, the input sequence of signalstates can be transformed into a matrix of scaling factors of the sameshape as the input, by a fully connected neural network (FCNN) 1810 withSoftmax activation 1820. This matrix represents the attentioncoefficients across signals and across states, effectively capturinginteractions across both domains. This matrix is combined with the inputstates through element-wise multiplication to produce the attention map1857. This modified attention layer can search over its input oftime-locked mini-sequences from different signals and identify importantpatterns within subsets of these signal types. These patterns, encodedin the heat map, are then sent to a multi-memory aggregator (such asshown in FIGS. 15 and 16) to maintain a longer temporal history. Theattention module also provides better interpretability through itsheatmap.

The activity A time window could be a different length than the activityB time window and can be different for different people. Normalizingdifferent activity can be done since each signal can be produced indifferent system with a different dynamic.

Correlating a person's activity with his/her high heart rate, high bloodpressure, and shortness of breath can be performed. This developmentalso covers how these correlation values over time help predict riskfactor and stages of heart failure a person can be expected toexperience if not followed up with a doctor. This development describesa new multi-level modified RNN realization that can learn risk factorsand predict possible heart failure based on all correlation patterns.

Signals of patients that have been diagnosed with different stages ofheart failure have been measured and multiple correlation time series(curves) have been computed. They can be presented simultaneously asinputs to the interconnected multi-level modified RNN architecture thatsends their outputs to the decision-making block 330 shown in FIG. 3.

The system 100 can use the outputs of the interconnected multi-levelmodified RNNs to map patients to a stage of heart failure they belong tobased on correlation patterns, features derived from them, and riskfactors learned in neural networks according to heart failureguidelines. For example, correlation of cardiac output and shortness ofbreath with activity of patient can be used to differentiate acongestive heart failure (CHF) patient from an athlete. One layer thatcan perform this kind of classification is the softmax layer 1820 asshown in FIG. 18, whose output represents the probability of the inputdata belonging to each of the four heart failure classes defined by theNew York Heart Association.

FIG. 17A is a block diagram of an embodiment of decision making thatoutputs a scalar. FIG. 17B is a block diagram of an embodiment ofdecision making that outputs a binary decision. FIG. 17C is a blockdiagram of an embodiment of decision making that outputs a multi-classdecision. In one embodiment, the decision making block 330 performs aclassification decision, such as heart failure classification. Thisdecision making can take three different forms, as described in FIGS.17A-17C.

The three forms may inclsude: 1) a positive scalar for risk assessment(to quantify a risk score between zero and 100, for example) as shown inFIG. 17A; 2) a binary format for prediction (yes/no) as shown in FIG.17B; or 3) a multi-class format for multi-class classification (heartfailure classes A, B, C, D) as shown in FIG. 17C. In all three cases, aninput 1710, 1720, 1730 of size 1×D can be passed through a one-layerfully connected neural network 1712, 1722, 1732 with weights of size D×kfor k classes (for a positive scalar, k=1); the output 1714, 1724, 1734,then, will be of shape 1×k. The differences between these threeapproaches are as follows. For risk assessment in FIG. 17A, k=1, and sothe output will be of size 1×1. This raw value can be used as thedecision. For binary prediction in FIG. 17B, the output can be of size1×2. The decision will be made from this array by using an Argmaxfunction 1726, which returns the index of the maximum element in thearray. So an output consisting of [0.3, 0.7] will return 2 for thedecision, since the maximum element occupies index 2. Multi-classclassification in FIG. 17C is similar to binary classification, where kcan be any integer greater than 2, and an Argmax function 1736 is usedto make the decision. Different decisions can be made simultaneously byfeeding the multi-signal memory output in parallel through severaldifferent layers. The decision-making block can be used to train thefull system end-to-end. After training is complete in the cloud 110, theparameters and models can be used to reconfigure the edge system 130,150.

In another embodiment, the decision making block can perform scatteringon the outputs of the MLM-RNNs. Scattering is the problem of dividing aset of data so that patients within each division are more similar toeach other than to those in other divisions. Using a combination ofmultiple bio-signals and sensor types may increase discriminative powerof a scattering algorithm. In one embodiment, as shown in FIG. 3,pre-trained RNNs 320, 322, 324 can be used to dynamically scatter theircross-correlation inputs to make these graphs. This allows each patientto occupy a different region of parameter space at different points intime; alerts can be set if the patient is detected to be in a differentsubspace of parameter space for some period of time, based on the recentand current scattered input.

A selection of three variables can be scattered to make interpretablegraphs for clinicians and patients to review. By storing the dynamicscattering across time, animations can be used to illustrate patientprogress or deterioration across time.

In one example, heart failure patients may be distinguished from healthycontrols by scattering blood pressure, activity, heart rate, and HRV.Heart failure patients are more likely to have high blood pressure andlower HRV compared with controls. They are likely to have a higher heartrate during periods of low to moderate activity, due to their heartworking harder to increase its effective output. In this example, RNN320 can take correlation output from blood pressure and HRV. Similarly,RNN 322 can take correlation output from heart rate and activity. Thesetwo correlations can be scattered to distinguish healthy vs. heartfailure patients.

In one embodiment, the system 100 can take output of the RNN as shown inFIG. 3, without cross-talk between cell states of the separate RNNs. Inanother embodiment, the system 100 can take output of the RNN as shownin FIG. 3 with cross-talk between cell states (as explained in FIG. 23).This allows the RNNs to share information with each other in order toincrease their utility as input for the scattering algorithm. In someembodiments, this can be conceptualized by treating the series of matrixmultiplications inside FIG. 23 as multiple linear transformations, whichcan function to project the input data into more easily separablesubspaces.

A few specific examples of what types of decisions can be made in thedecision making block 330:

1) Binary output as prediction of high probability of adverse event,such as acute heart failure.

2) Risk Score, a number between 1 and 100 that quantifies the patient'scurrent overall risk. This prediction is made by passing through a1-unit neural network with scaled sigmoid activation function.

3) Heart Failure Classification (NYHA): The New York Heart Associationdivides heart failure into four classes: Class 1, 2, 3, and 4, based onlevel of activity and presence of other symptoms. Routing themulti-signal memory through a fully-connected layer with four units (onefor each class) and Softmax activation can predict the classification.The output of Softmax represents the probability of the input databelonging to each class.

4) Heart Failure Classification (ACC): The American Heart Associationand the American College of Cardiology have developed classificationtypes A, B, C, and D based on structural heart disease and presence ofheart failure symptoms. Routing the Multi-signal memory through afully-connected layer with four units (one for each class) and Softmaxactivation can predict the classification. The output of Softmaxrepresents the probability of the input data belonging to each class.

5) Heart-failure subtypes: Softmax activation over N classes

a) As another objective, a separate model(s) can be trained to classifythe patient into one of several sub-types of heart-failure. Theseinclude reduced vs. preserved ejection fraction.

b) An alternative model can be trained to distinguish between left-sidedand right-sided heart failure, and identify congestive heart failure.

Then in the edge, the parameters of a pretrained interconnected neuralnetwork can be optimized in real time with semi-supervised learningschemes Additionally, teams of trained clinicians can help to provideannotations on data from each patient in order to fine-tune andpersonalize each patient's own machine learning models.

In some embodiments, Equation 2 shown below provides a high correlationof heart rate (HR) and systolic blood pressure (BP) detected in realtime on the edge to manifest oxygen demand exceeding oxygen supply andit can predict myocardial ischemia or myocardial infarction especiallywhen it has correlation with reduced activity.

C(n)=Σ_(m=n−w+1) ^(n)HR_((m-k))SBP_((m))  Equation 2

where ‘w’ is the shorter time window for the two activities that havepassed their corresponding thresholds and have triggered execution ofcorrelation between the two.

Some risk factors that are genetic risk factors plus environmental riskfactors accumulated over time are quantified. These bio-markers can bediscovered for different genetic pools. If one has some genetic datafrom some patients and can correlate some of these bio-markers withgenetic data, then one can establish a reference data set and parametersof a neural network model that helps to predict that people with thosebio-markers may have genetic background of a given disease. The systemand method can suggest to a patient (insurance) to take genetic testingto confirm diagnosis and start a right treatment early on.

Discovering these bio-signals dependency patterns gives new insights todoctors that helps not only early diagnosis of existing diseases, andnarrowing down and identifying a source, but also discovering newdiseases and selecting a right treatment plan based on quantitativepatterns of data (evidence-based medicine and precision medicine andpersonalized medicine).

The system 100 in FIG. 1 and FIG. 4C is capable of facilitatingbiofeedback. Detecting dependency patterns of a user's mood and heartrate and sensing the mood and commanding to a multi-media center to playmusic and/or play a video that regulates the user's mood and sense andfeedback to a controller (multi-media player) to feed right input to oursensory modality (ear, eye) can be performed. If this control loop isclosed, the system and method can be used to help the user's heart rateto come down, avoid burning out the heart muscles and eventually heartfailure or heart attack.

In certain embodiments, each disease/condition can be configured usingan efficiently designed multi-level correlation algorithm in a networkthat specializes in a self-configured multi-level interconnectedmodified RNN as multiple measured time series are presented to thenetwork in real time. One embodiment could be a hardware realization toget the best speed and power consumption for a health managementapplication of the edge technology such as the prediction of Acute HeartFailure.

In some embodiments, the system 100 takes as its input one or moremeasurable bio-signals from wearable or implanted sensors. Thesebio-signals may include, but are not limited to: ECG, activity, bloodpressure, SpO₂, respiration, bioimpedance, and body weight.

A variety of algorithms allow the edge to provide remote patientmonitoring combined with personalized medicine. This includes adverseevent detection and prediction.

Multi-Modality

There are different kinds of recurrent neural networks (RNNs) 320, 322,324 as described with respect to FIG. 3. One kind of RNN called longshort term memory (LSTM) is modified in two forms as describedfollowing:

In one embodiment, by modifying the LSTM to allow two input sources,each having a different sampling time, a multi-modal LSTM (MM-LSTM) wasdeveloped. This approach is described in conjunction with FIG. 9 andFIG. 21.

Another embodiment is implemented by applying each input from a givensource (or derived features) to a single LSTM within a bank of LSTMs,and then applying attention on states of a multi-level LSTM (MLM-LSTM).This MLM-LSTM encompasses a bank of LSTMs. This approach is described inconjunction with FIG. 10 and FIG. 15 (FIGS. 11, 12, 13 and 14 depictdifferent embodiments of MLM-LSTM).

FIG. 5 is a block diagram of another embodiment for processing humanphysiological signals through decision making such as performed on thesystem of FIG. 1. Inside the edge system 130, 150, features can beextracted from multiple sensors as shown in FIG. 5. Feature extractionsubsystems 510, 512, 514 compute features from sensory signals as shownin the figure.

In FIG. 5, features can be extracted from various sensors. Thesefeatures are passed into various blocks, whose outputs are integrated bya decision maker 540. In one embodiment of the multi-level modified deeplearning approach shown in FIG. 5, derived features first go throughsub-blocks such as a LSTM 520, 522, 524. In this embodiment, these maybe “vanilla” LSTMs.

LSTMs provide an improvement over traditional “vanilla” recurrent neuralnetworks by allowing continuous regulation of the cell memory throughvarious gates. It also helps mitigate the problems of vanishing andexploding gradients during back-propagation.

LSTM Gates Let:

h_(t-1)=previous hidden state

W_(g)=recurrent matrix through gate g

b_(g)=bias through gate g

Forgetting gate:

-   -   i. f_(t) (x)=σ(W_(f)[h_(t-1),x_(t)]+b_(f))

Input gate:

-   -   i. i_(t)(x)=σ(W_(i)[h_(t-1),x_(t)]+b_(i))

Tan h layer:

-   -   i. {tilde over (C)}_(t)=tan h(W_(C)[h_(t-1),x_(t)]+b_(c))

Output gate:

-   -   i. o_(t)(x)=σ(W_(o)[h_(t-1),x_(t)]+b_(o))

The Tan h layer merges the two paths into a shared cell state.

The derived features are passed into a first LSTM to learn patterns fromeach of sensor features and then go through another MM-LSTM 530, 532 (asdescribed below) to learn patterns on combination of two or moresensors' features (530), plus inputs from the state of the first groupof LSTMs.

The output of first group of LSTMs and the second group of LSTMs arecombined in a late fusion block that performs decision making (540). Tocombine the output of all previous LSTMs into one input, the outputstate vectors from each LSTM can simply be concatenated into a longervector. This vector can then be passed through a single fully-connectedlayer to make a decision. Different kinds of decisions are shown in FIG.17.

The complete description of decision making blocks are described aboveas part of FIG. 3 description. A brief description is provided below.

This decision maker 540 can also take as its input a group ofpredictions in order to decide based on a weighted vote count, or otherfeatures such as transformed time-series data which is mapped through aSoftmax function to a predicted class (this can give a probability ofbelonging to class 2 or class 3 heart failure, based on data of lasttwelve hours or last one hundred twenty states in the LSTM bank).

To perform vote counting, the previous layers (520, 522, 524, 530, 532)may use, e.g., a Softmax layer to map their output directly to aprediction. Then the decision maker 540 can hold a vote over thesedecisions to determine the majority, and report the majority as itsdecision.

For risk assessment, the previous layers can compute a scalar output,and the decision making block 540 can return a weighted average as thefinal quantification of risk.

Multi-Modal LSTM (MM-LSTM) The MM-LSTM can incorporate two or moreinputs from different modalities. In one embodiment, two separate pathsare used for each modality of incoming data and each modality is treatedwith different weight matrices.

Let.

u_(t)=[x_(t); y_(t)].

-   -   h_(t-1) ^(x)=previous hidden state for x    -   h_(t-1) ^(y)=previous hidden state for y    -   W_(g,x)=recurrent matrix for x through gate g    -   b_(g,i)=bias for x through gate g        -   The two separate paths are routed through the following            junctions:        -   Forgetting gates:            -   i. f_(t)(x)=σ(W_(f,x)[h_(t-1) ^(x),x_(t)]+b_(f,x))            -   ii. f_(t)(y)=σ(W_(f,y)[h_(t-1) ^(y),y_(t)]+b_(f,y))        -   Input gates:            -   ii. i_(t)(x)=σ(W_(i,x)[h_(t-1) ^(x),x_(t)]+b_(i,x))            -   iii. i_(t)(y)=σ(W_(i,y)[h_(t-1) ^(y),y_(t)]+b_(i,y))        -   Tan h layer:            -   ii. {tilde over (C)}_(t)=tan h(W_(Cx) [h_(t-1)                ^(x),x_(t)]+b_(cx))+tan h(W_(cy)[h_(t-1)                ^(y),y_(t)]+b_(cy))        -   Update:            -   C_(t)=mean(f_(t)(x),f_(t)(y))*C_(t-1)+{tilde over                (C)}_(t)        -   Output gates:            -   ii. o_(t)(x)=σ(W_(o,x)[h_(t-1) ^(x),x_(t)]+b_(o,x))            -   iii. o_(t)(y)=σ(W_(o,y)[h_(t-1) ^(y),y_(t)]+b_(o,y))        -   The Tan h layer merges the two paths into a shared cell            state.

FIG. 21 is a block diagram of an embodiment for a portion of an examplemulti-modal (MM) long short-term memory (LSTM) showing the fusion ofLSTM states as input, and signals as separate input. The MM-LSTM canfacilitate multiple different inputs. Through a forward pass of thedeveloped MM-LSTM, two inputs x and y are kept independent for most ofthe path. A separate hidden state is maintained for each of the separateinputs. These are maintained separately by routing through unique setsof neural network weight matrices.

To determine the amount of information to forget from the cell state2110, the forgetting coefficients are first computed for each input.These are each in the range of zero to one. They are then routedtogether through a mean block, which computes the average scalingcoefficient with which to multiply the cell state 2110. This result ismultiplied by the cell state element-wise to scale each entry by thesame factor between zero and one. For updating the cell state 2110through addition, the combined output of a tan h layer from eachseparate path are added to the cell state. The result is a new cellstate 2120, to be used for the next time step. This cell state is routedthrough a tan h block, and this output is multiplied with each outputgates to yield the new updated, separate hidden states. As anotherembodiment, the system may use the state of single LSTMs that areoperating on a single feature. Some of the inputs to a MM-LSTM can bethe hidden state from other LSTMs, such as from LSTMs 520, 522, 524shown in FIG. 5.

FIG. 22 is a block diagram of an embodiment for a portion of an exampleMM-LSTM showing how both signals and hidden states from other LSTMs canbe combined inside of the MM-LSTM. FIG. 22 shows a cell state 2210 andanew cell state 2220 and also shows how both signals and hidden statesfrom other LSTMs can be combined inside of the MM-LSTM, such as inMM-LSTM 530, 532 described with respect to FIG. 5. This embodiment canprovide a significant innovation over “vanilla” LSTMs by combiningmultiple raw signals from different modalities with the changing statesof other LSTMs entrained on the dynamics of those signals. Additionalinputs in FIG. 22 are routed through the same mean and summation blocksas in FIG. 21. The output from output gates is multiplied with the sametan h(Ct) as in FIG. 21. These networks are used in components 530, 532shown in FIG. 5, 640, 642 shown in FIG. 6, 740, 742 shown in FIG. 7,830, 832 shown in FIG. 8, and 960, 962, 964 shown in FIG. 9. Thisdistributed training may happen on the edge system 130, 150 and thecloud 110 for supervised learning shown in FIG. 1.

For a machine learning model with many input features, or a largemulti-dimensional input, it is often useful to prune the number offeatures that are fed into the model. This can save processing time andstorage.

One method of feature selection can be described as recursive featureelimination (RFE). For a dataset with N features, RFE tries to find asubset of k<N features that yield a validation accuracy within somethreshold of the accuracy obtained by using the full feature set.

For each model of size m, where k<m<=N, the features are rankedaccording to their importance, or their contribution to model accuracy.The least important feature is removed, and the model is trained againon m−1 features. This process is repeated until only k features remain,or until validation accuracy falls below threshold.

In some embodiments, for a given task or objective by using anappropriate feature selection scheme, the features that gain highestcorrelation with labels on associated data samples are selected. Afterthe feature selection for a given task or objective, the cloud 110 maychange model configuration in the edge system 130, 150 based on a givenobjective. These on-the-fly reconfigurable models allow our technologyto offer multiple objectives and services for health care, including butnot limited to acute heart failure prediction, myocardial infarctionprediction, arrhythmia detection, orthostatic hypertension detection,etc.

FIG. 6 illustrates another embodiment of a novel machine learningapproach where the system 100 incorporates a scheme to learn over anextended period in time. In this embodiment a correlation sub-system620, 622, 624 is utilized along with multi-level modified LSTMs 630,632, 634, 640, 642, which work together to summarize longer sequencesinto a decision.

A feature extraction sub-system 610, 612, 614 computes features fromsensory signals received from Sensor 1 to Sensor X. For example, heartrate variation can be computed from an electrocardiogram (ECG) signalsensed from the heart, and sitting or walking and a number of steps canbe computed from X, Y, Z acceleration signals as a feature representingphysical activity.

The correlation sub-system 620, 622, 624 computes correlation of twofeatures derived from one or more sensory signals. Each pair of signalsmay be routed to one of the correlation blocks 620, 622, 624, whosecorrelation signal outputs are sent to an LSTM 630, 632, 634.

For example, congestion of the lungs can be extracted from signalsrecorded by a thoracic bioimpedance sensor. Correlation sub-systems 620,622, 624 may estimate correlation between two signals, such as drops inHRV and lower activity; more congestion and higher percentage of AFIB;more congestion and lower activity; lower oxygen saturation level withlower physical activity. These high correlations may be bio-markers thathelp to predict increasing risk of acute heart failure.

FIG. 6 takes the Multi-Level Modified deep learning approach where eachcorrelation output of pair of candidate features go through a first LSTMgroup 630, 632, 634 to learn a pattern on one of bio-markers and then gothrough an MM-LSTM 640, 642 to learn patterns on combinations of two ormore of bio-markers plus inputs from a state of the first group of LSTMs630, 632, 634. The output of first groups of LSTMs and the second groupof LSTMs may be combined in a late fusion block that may performdecision making 650. This block is identical to decision making 540shown in FIG. 5. The different kinds of decisions are described inconjunction with FIG. 17. This distributed training happens on the edgeand the cloud for supervised learning.

FIG. 7 is a block diagram of another embodiment for processing humanphysiological signals through decision making such as performed on thesystem of FIG. 1. The block diagram depicted in FIG. 7 is functionallysimilar to the one illustrated in FIG. 6 with the exception of specificexamples of signal-processing blocks for each feature extraction module610, 612, 614. The remaining portions of FIG. 7 are functionally similarto that of FIG. 6.

In some embodiments, a peak detection block 710 uses an algorithm thatfinds the R peak from the QRS component of an ECG signal, and returns apeak-to-peak interval referred to as a R-R interval. HRV can be computedfrom statistical properties of R-R intervals. Physical activityrepresents an important metric of daily health. A block 712 can take asits input a signal from an accelerometer sensor and summarize the datainto various measurements of physical activity, such as step count (thenumber of steps taken by the patient) or calories burned, within sometime interval. These measurements of physical activity can be computedonboard the sensor, or computed by the block 712 using a convolutionalneural network (CNN) or other machine learning model (such as randomforest, etc.) to process the raw 3-axis signal from the accelerometer.

One particular scheme to process the raw accelerometer X, Y, Z data isto first use a random forest to classify the type of activity from thesevalues. These types of activity include sitting, lying down, walking,etc. When walking is detected, the number of steps are found by countingthe number of peaks within the smoothed signal. Steps are usually mostprominent along the Z axis, depending on sensor orientation. SpO₂ ismeasured from light sensors on the skin. A feature block 714 can take asits input the detected SpO₂ signal from a wearable sensor and relay thesignal forward to the correlation sub-system. Correlation blocks720-724, LSTMs 730-734, MM-LSTMs 740 and 742, and a decision makingblock 750 are similar to those of FIG. 6.

FIG. 8 is a block diagram of another embodiment for processing humanphysiological signals through decision making such as performed on thesystem of FIG. 1. The block diagram depicted in FIG. 8 is similar to theone illustrated in FIG. 7. Features detected at blocks 810, 812 and 814are passed directly into respective LSTMs 820, 822 and 824. This makesit functionally similar to that illustrated in FIG. 5 with the exceptionof specific examples of signal-processing blocks for each featureextraction module 510, 512, 514. MM-LSTMs 830 and 832, and a decisionmaking block 840 are similar to those of FIG. 7.

In the configuration of FIG. 8, pairs of features are sent together intoan MM-LSTM 830, 832. In one embodiment of a MM-LSTM, one separate pathis used for each modality of incoming time series data. These separatesignal types are merged in the model at the time the cell state of theLSTM is updated. The gate functions are modified to be able to inputmore than one time series or features extracted from more than one timeseries to a single LSTM to combine learning from patterns on bothsignals. Any significant changes in two signals that are related orcorrelated to each other but may have a lag respect to each other areidentified.

FIG. 9 is a block diagram of another embodiment for processing humanphysiological signals through decision making such as performed on thesystem of FIG. 1. Figures shows one implementation of feature extractionand machine learning algorithms. In the arrangement illustrated in FIG.9, pairs of features are passed into various blocks, whose outputs areintegrated by a decision maker 970. The types of features may include,but are not limited to, an arrhythmia percentage 910, a heart ratevariability (HRV) 912, physical activity 914, and blood oxygensaturation SpO₂ 916.

An arrhythmia detector 910 performs detection of cardiac arrhythmias.The arrhythmia detector 910 may include a CNN, whose output is analyzedby an LSTM, and an attention layer which processes the LSTM output.First, the CNN searches for certain features, like heart-beat frequencyand shape, across the length of the ECG signal. The CNN can also providetemporal downsampling to the signal via pooling layers. The output ofthe CNN represents compressed temporal features over time. These are fedto an LSTM which is well-suited for time-series analyses. Finally, thereturned sequences from the LSTM are fed through an attention layer,which performs multi-class classification through a Softmax layer.

The attention layer helps the model to be more interpretable, byproviding a visual indication to the medical provider that highlightsthe relative contribution of each segment of the input signal to theclassification decision made by the model.

Another way to increase interpretability of machine learning models isby using a technique called class activation mapping (CAM). A classactivation map for a particular category or class indicates thediscriminative regions of the input signal used by the model to identifythat category. In the case of an ECG signal, for instance, this wouldshow which portions of the ECG trace were most influential in leading tothe prediction of a certain arrhythmia class.

For each segment of an incoming ECG signal, the arrhythmia detector 910can output a number that represents the proportion of that segmentcontaining arrhythmia. For example, for a 10-beat segment with a singlepremature ventricular contraction, the percentage of PVC would be 10%. Asequence of these arrhythmia proportions can be combined with anothersignal using a correlation block, such as illustrated in FIGS. 9 and 10,or combined with another signal directly into an MM-LSTM as illustratedin FIG. 9. Alternatively, these proportions are used alone astime-varying input into one of a bank of LSTMs as illustrated in FIG.10.

A peak detection block 912 is identical in input, processing, and outputto earlier instances of the peak detection blocks 710 of FIGS. 7 and 810in FIG. 8. A physical activity block 914 is identical in input,processing, and output to earlier instances 712 of FIGS. 7 and 812 inFIG. 8. A SpO₂ block 916 is identical in input, processing, and outputto earlier instances 714 of FIGS. 7 and 814 in FIG. 8.

Each pair of the signals from the feature blocks 910, 912, 914, 916 isrouted to a correlation block 940, 942, 944, whose correlation signaloutput is sent to an LSTM 950, 952, 954. These pairs are also senttogether into an MM-LSTM 960, 962, 964. In one embodiment of MM-LSTM,one separate path is used for each modality of incoming time seriesdata. These separate signal types are merged in the model at the timethe cell state of the LSTM is updated. The gate functions of the LSTMare modified to be able to input more than one time series or featuresextracted from more than one time series to a single LSTM to combinelearning from patterns on both signals. Any significant changes in twosignals that are related or correlated to each other but that may have alag respect to each other are identified. This functionality isillustrated and described in conjunction with FIG. 9, FIG. 21 and FIG.22. Both the LSTM and MM-LSTM outputs are sent to the decision maker 970that is further described in conjunction with FIG. 17.

Another embodiment we have labeled as a multi-level modified LSTM or MLMRNN. In this embodiment, a separate LSTM is used for each modality, andthe modalities are used in the following manner. An Attention layerattends over consecutive cross-modality cell states. This Attentionlayer has an elastic mechanism to aggregate different data-rate inputsand capture correlation between different time intervals of inputshaving a wide attention strip (2-D attention heat map). The output isfed into a gated memory to store a history of cross-modalityinteractions. This is described in more detail in conjunction with FIG.10 (also in FIG. 3 for the MLM RNN).

In the embodiment of FIG. 9, the HRV of the peak detection block 912 andthe step count of the activity block 914 may need to be buffered in ananomaly buffer & delay block 920 and aligned before passing into thecorrelation blocks. For activity detection, an attention block 922 isused to identify regions of low activity. This block 922 may be used tofind important segment(s) of an input signal, along with aclassification decision to categorize the input signal. The new outputis used to locate certain patterns in the input signal in order tohighlight the corresponding time-steps for downstream blocks. In anexample, step counts or activity type are used as an input, and producea heat map of detecting decrease in activity as a new output of theattention block 922 to feed to MM LSTM 960 and correlation block 942. Abuffered HRV signal, along with the attention map, are passed into acorrelation block 942. Additional destinations for the attention map arecorrelation block 940, which computes the cross-correlation witharrhythmia percentage; and the MM-LSTM 962 which combines it with thebuffered HRV 920.

As incoming signals are recorded, they are saved in a rolling buffersuch as an anomaly buffer & delay block. This enables the system toalways keep a recent signal history ready for computation. The rollingbuffer starts out empty, and begins filling with incoming data byconcatenating new samples onto the end. When the buffer reaches apredetermined length, the oldest values inside of the buffer areremoved, and the remaining values are shifted to allow room for the newvalues to append onto the end of the most recent values.

In some embodiments, the system 100 can detect when patient vital signsdeteriorate (anomaly detection). One way that this is achieved is byusing threshold detectors or a modified attention network describedbelow or by arrhythmia detection. These anomaly detection blocks 920,922 function as a switch that starts a sequence of events when the inputsignal is determined to cross a certain threshold. Example thresholdsmay include, but are not limited to: resting heart rate above about 100bpm, SpO₂ falling below about 85% or a drop of more than about 10 pointsin a short time, and HRV decrease before activity decrease.

There are many bio-signals whose values can be used to directly decodepatient status. Examples include heart rate variability and physicalactivity. In some cases, a state change indicated by one changing signalcommonly precedes another state change in a separate bio-signal by someinterval. An example is a decrease in HRV hours before a decrease inphysical activity. For these two signals, their cross-correlationrepresents an important feature describing their temporal interactions.The modified attention block as designed by the developer helps tofirst, detect lower physical activity state and second, run crosscorrelation in the most efficient way to save processing power and powerconsumption on the edge system 130, 150. This allows the edge to be asmaller size and portable for outdoor use (so as connect to a cellularIOT network) and lower cost.

Two options may be used for the correlation blocks as follows:

-   -   Option 1: For a given pair of signals, if a threshold-crossing        is triggered for one of the signals, that signal is routed along        with a separate signal into a correlation block. Then,        cross-correlations are computed for the pair of signal buffers.    -   Option 2: For a given pair of signals, if a threshold-crossing        is triggered for one of the signals, it registers active as an        event and will wait for another signal        threshold-crossing-detector. When both signals have crossed        their corresponding thresholds, the correlation block becomes        active. Then, cross-correlations are computed for the pair of        signal buffers.

A lag can be computed in some cases based on simple threshold crossingblocks applied on both inputs to a correlation block. The correlationblock takes as input the two input signals and the time-lag to output anew representation of the two signals, aligned at the time of the secondsignal threshold crossing. After aligning these two inputs, they areinput not only to the correlation block 942 but also to the multi-modalLSTM block 962 that may learn some cross-signal interactions.

The time-varying correlation signal can be fed as input to an LSTM 952that is interested in the time-course of the correlation output itself.For example, sharp (smooth) peaks in the correlation signal represent amore transient (long-lasting) correlation in time. This signal can beused alone by an LSTM 920, 930.

As shown in FIG. 9, the ECG signal goes through the peak detection block912 to detect the R peak in QRS portion and consequently compute the RRinterval which is used to derive HR and HR variation (HRV). Someembodiments use a bi-directional LSTM to detect R peaks; howevertraining of this LSTM can happen as a subtask before training of themain model for the main task or objective such as acute heart failureprediction based on the multi-modal LSTM 960, 962, 964, 970 of FIG. 9 orthe multi-level LSTM 1040, 1050, 1060 to be described in FIG. 10.

Some signals may contain important features which cannot be detected bysimple threshold crossings. This presents a challenge for properlyaligning two input signals to an LSTM for the purpose of findinginteractions. In the solution utilized by the system 100, the attentionblock 922 is utilized for finding regions of interest of its inputsignal.

For some time-varying features, like physical activity, some embodimentsexploit the capabilities of the attention layer to both categorize theinput signal into one of multiple classes, and to return a time-varyingsignal that represents a heat map over time on the input. This heatmapcan be used as an input to correlation block as in the correlation block940, or as an input to an LSTM 962 combined with other physiologicalsignals. This represents a novel way of using attention weights,representing a heatmap, in a manner that combines the extractedcharacteristics of one signal with other raw signals.

Each type of vital sign represents a unique view into the patient'scurrent health status. These vital signs each have their own individualrepresentation space and dynamics. From a collection of vital signs ofseparate modalities (e.g. heart rate, heart rate variation, BP, SpO₂,physical activity), some of the signals may share some mutualinformation due to dynamics of interactions between different organssuch as the heart, lung and nervous system, while containing someseparate independent information about the patient's current status. Toachieve a more complete view of the time course of patient condition,multiple modalities are combined in some modules, such as thecorrelation block 940, 942, 944 and MM-LSTM 960, 962, 964.

Finally, all engineered outputs and predictions are combined andinterpreted by the decision maker 970. The different kinds of decisionsare described in conjunction with FIG. 17.

FIG. 10 is a block diagram of an embodiment for processing humanphysiological signals through decision making and including an exampleattention network and an example multi-signal memory aggregator such asperformed on the system of FIG. 1. Referring to FIG. 10, memory fusionlayers are described. Due to the different time course of separatebio-signals, and the different kinds of temporal features they maycontain, it can be confusing for a single LSTM to make sense of inputdata including several distinct signals. To achieve a union of signalanalysis across different types, some embodiments extend a memory fusionnetwork to operate on a combination of bio-signals in the architectureshown in FIG. 10.

In certain embodiments, the arrhythmia detection block 1010 is identicalto the arrhythmia detection block 910 described with respect to FIG. 9.The peak detection block 1012 is identical to the peak detection block912 described with respect to FIG. 9. The activity detection block 1014is identical to the activity detection block 914 described with respectto FIG. 9. In this embodiment, blood pressure 1016 is used instead ofSpO2 916. The anomaly buffer & delay 1020 is identical to the anomalybuffer & delay 920 described with respect to FIG. 9. The attention block1022 is identical to the attention block 922 described with respect toFIG. 9. The correlation block 1030 is identical to the correlation block940, and the correlation block 1032 is identical to the correlationblock 942, described with respect to FIG. 9. The correlation block 1034only differs from the correlation block 944 of FIG. 9 by using bloodpressure instead of SpO₂. The system 100 takes as its input one or moremeasurable bio-signals from wearable or implanted sensors. Thesebio-signals may include, but are not limited to: ECG, activity, bloodpressure, SpO₂, respiration, bioimpedance, and body weight.

First, each signal type is routed alone to its own LSTM 1040 within abank of LSTMs. Pairs of signals are also routed into correlation blocks1030, 1032, 1034, whose outputs are each sent to one of the LSTMs 1040.The group of LSTMs 1040 processing separate signals is consideredcollectively as a bank. A history of states from each of these LSTMs1040 is collected, and analyzed by an attention network 1050. The outputof this attention network learns interactions across time and acrosssignals. A history of these interactions is summarized using aMulti-signal memory aggregator 1060.

In this embodiment, the bank of LSTMs and the attention network 1050work together as an encoder, selecting the relevant information to passto the next layer. The Multi-signal memory aggregator 1060, then, worksas a decoder to help generate a prediction from the states output by theencoder. A decision maker 1070 makes the final decision by transformingthe output of the multi-signal memory aggregator 1060, similar to thedecision maker 540 described with respect to FIG. 5, but without needingto concatenate multiple vectors into one.

FIG. 11 is a block diagram of another embodiment for processing humanphysiological signals through decision making and including theattention network and the multi-signal memory aggregator such asperformed on the system of FIG. 1. The sub-system depicted in FIG. 11 isidentical to that illustrated in FIG. 10, except that generic signalsare routed through unspecified feature extraction modules. Thisconfiguration shows how an arbitrary combination of features extractedfrom a collection of sensors 1110, 1112, 1114, 1116 can be routedthrough anomaly detection blocks 1120, 1122, 1124, 1126 and then sentthrough LSTMs 1140 and MLM-LSTMs 1150 before or after combining withanother feature through cross-correlation via correlation blocks 1130,1132, 1134. Unprocessed or cross-correlated features are passed througha bank of LSTMs 1140. The remaining signal paths through components1150, 1160, 1170 are functionally identical to those depicted incomponents 1050, 1060, 1070 in FIG. 10.

FIG. 12 is a block diagram of another embodiment for processing humanphysiological signals through decision making and including theattention network and the multi-signal memory aggregator such asperformed on the system of FIG. 1. The sub-system depicted in FIG. 12 issimilar that illustrated in FIG. 10, but the input to each LSTM 1240 inMLM_LSTM includes only the outputs from correlation blocks 1230, 1232,1234, 1236, 1238. The remaining blocks 1210, 1212, 1214, 1216, 1220,1222, 1250, 1260 and 1270 are similar to those of FIG. 11.

FIG. 13 is a block diagram of another embodiment for processing humanphysiological signals through decision making and including theattention network and the multi-signal memory aggregator such asperformed on the system of FIG. 1. The sub-system depicted in FIG. 13 isthe same as that illustrated in FIG. 12, but here the LSTM blocks 1240are replaced with RNNs 1340. This is used to illustrate how the bank ofLSTMs can be replaced by a bank of any recurrent neural architecture.The remaining blocks 1310, 1312, 1314, 1316, 1320, 1322, 1330, 1332,1334, 1350, 1360 and 1370 are similar to those of FIG. 12.

FIG. 14 is a block diagram of another embodiment for processing humanphysiological signals through decision making and including theattention network and the multi-signal memory aggregator such asperformed on the system of FIG. 1. The sub-system depicted in FIG. 14 isidentical to previous examples of MLM_LSTM except that it eliminates thecorrelation blocks altogether and includes an SPO2 block 1418. Theremaining blocks 1410, 1412, 1414, 1416, 1420, 1422, 1440, 1450, 1460and 1470 are similar to those of FIG. 13.

Memory Fusion: More Details

Each of the different input signals is first passed separately into oneof the LSTMs 1040 within a bank of LSTMs. Each LSTM in this bank canlearn temporal features for a specific signal type. Utilizing a bank ofseparate LSTMs allows each type of signal to have a different input,memory (cell state), and output shape, which provides flexibility forcombinations of signals with different sample rates.

FIG. 15 is a block diagram of the example attention network andmulti-signal memory aggregation in more detail. FIG. 15 shows themodified attention network and the multi-signal memory aggregation inmore detail. The cell states from the bank 1540 of LSTMs are bufferedand passed into the attention network 1550, which finds interactionsacross the different signals and across time; these interactions arereflected in each of the two dimensions of its output. To compute theattention output X, the input sequence of signal states is transformedinto a matrix of scaling factors of the same shape as the input, by afully connected neural network (FCNN) 1510 with Softmax activation 1520.This matrix represents the attention coefficients across signals andacross time, effectively capturing interactions across both domains.This matrix is combined with the input states through element-wisemultiplication to produce the attention map 1557, the cross-signalinformation used as input for the multi-signal memory aggregator 1560.The multi-signal memory aggregator 1560 can include sub-blocks such asattention map for timestep 1562, 1564 and 1566.

The modified attention layer over consecutive cross-modality cell statescan be defined as:

$X_{s,t} = {B_{t} \odot \frac{\exp( a_{s,t} )}{\Sigma_{s^{\prime},\tau}\mspace{14mu}{\exp( a_{s^{\prime},\tau} )}}}$

where

-   -   X=attention map    -   B_(t)=cross-modality cell-state buffer    -   α_(s,t)=W_(α)*B_(t) (W_(α)=FCNN 1510)

This modified attention layer can search over its input of time-lockedmini-sequences from different signals and identify important patternswithin subsets of these signal types. For example, as described withrespect to FIG. 10, during the attended window 1050, systolic BP 1016could be decreasing while arrhythmia percentage 1010 is increasing. Theattention layer 1550 can learn that this combination is especiallyrisky, and weight the appropriate elements of its output accordingly.

The strategy of attention over states can be applied to any collectionof elements within a bank of recurrent neural networks. These elementscould be from any type of recurrent neural architecture, includingvanilla RNNs, LSTMs, or others. In the case of vanilla RNNs, theseelements are hidden states, while in the case of LSTMs, these elementscould be hidden states or cell states. In either case, attention can beapplied to a buffer of states. FIG. 18 illustrates a way that attentioncan be applied to various recurrent neural architectures.

Referring back to FIG. 15, the cross-signal information from attentionis stored over time in the multi-signal memory aggregator 1560, amodified LSTM that updates its own state as a function of the attentionoutput and its own stored memories. This model can capture a longerhistory of interactions across signal types and across time. The outputof the multi-signal memory aggregator 1560 is fed into a decision makingblock 1570 for prediction.

FIG. 16 is a block diagram of an example attention map at a particulartime step and multi-signal memory aggregation in more detail. FIG. 16shows example components 1600 of the multi-signal memory aggregator 1560of FIG. 15. The multi-signal memory aggregator may include three gates.An input gate simply passes the input 1610 of the attention map for atime step through a neural network layer with linear activation toconstruct a proposed update to the internal cell state. An update gateinvolves a separate neural network with sigmoid activation whose outputdictates how much of the information in the proposed update toincorporate into the cell state at the next time step. The update andinput gates are depicted in a dashed block 1620. A retention gatedepicted by a dashed block 1630 involves another neural network withsigmoid activation whose output is used to control the amount ofinformation to maintain from the previous cell state itself. Each ofthese gates can take as their input the output from the attention block1610. The output from each of these gates is summed in a dashedsummation block 1640. This sum Ct is passed to a decision making neuralnetwork 1670 and is also stored in a memory 1650 for a next time step.

The multi-signal memory aggregator 1560 can store a history ofcross-modality interactions based on the following definitions:

1. Update Gate:

-   -   b. u_(t)=W_(i)(X)⊙σ(W_(u)(X))    -   c. X: input from attention    -   d. W_(i): Fully-connected neural network (FCNN)    -   e. W_(u): FCNN

2. Retention Gate:

-   -   a. r_(t)=σ(W_(r)(X))⊙C_(t-1)    -   b. W_r: FCNN    -   c. C_(t-1): Cell state from previous time step

3. Update Rule:

-   -   a. C_(t)=u_(t)+r_(t)

The final prediction can be made by taking the output from themulti-signal memory aggregator 1560 and passing this information througha decision making neural network 1570, 1670. Different decisions can bemade simultaneously by feeding the multi-signal memory output inparallel through several different layers. The different kinds ofdecision making are described in conjunction with FIG. 17.

Model Training Flowchart

The flowchart illustrated in FIG. 19 depicts a process 1900 of training,evaluating, and deploying the machine learning models for the system100. First, an initial architecture is chosen for the particular modelbased on the desired objective, such as objective i. Then, an annotatedoffline dataset is randomly divided into separate training andvalidation subsets. These sub-steps are included at a step 1910. Beforetraining, the model hyperparameters are chosen (for example, size ofeach training batch, learning rate, regularization penalties, etc.) at astep 1915 and the training of model j begins at a step 1920.

“Model j” refers to the model during a particular phase of architectureand hyperparameter optimization. At a training step 1925, the modelmakes a prediction on a training batch 1930 and a validation batch 1935.The output is compared with ground truth labels for that batch, and aloss is computed for that batch with respect to both the training 1940and validation 1945 batches. The validation set is not used for updatingmodel parameters, but only to monitor training progress to evaluate howwell the model generalizes on unseen data. The training and validationlosses are monitored together to evaluate model overfitting. Forexample, a high validation loss with a low training loss often signifiesoverfitting to the training set, meaning that the model will notgeneralize well to unseen data. Using a machine learning algorithm likegradient descent, the model parameters are updated as a function oftraining loss, with the goal of decreasing the loss for the nexttraining iteration. This training cycle of predict-compare-updatecontinues until either the training loss converges as determined at adecision step 1950, or the validation loss stops decreasing asdetermined at a decision step 1955. If the training loss converges, thefinal validation accuracy is compared at a decision step 1960 with apre-determined threshold for the particular task. If the validationaccuracy is too low, a new set of hyperparameters are chosen using amethod such as Bayesian optimization. Hyperparameters are alsoreconfigured if the validation loss stops decreasing.

When the final validation accuracy is high enough, the entire model isstored in the cloud at a step 1970. Then at a step 1975, for eachsubmodule (for example, FIG. 10 contains many LSTMs), a determination ismade by a decision step 1980 as to whether the submodule will fit inmemory on the edge device 1030, 1050. If the submodule requires too muchmemory or processing power or high computational time (inference time),it is designated for cloud computation at a step 1990. Otherwise, if itis suitable for deployment on the edge device, then that particularmodel is updated on the edge at a step 1985. In some embodiments, alledge system may communicate their available resources to the cloud.

Skilled technologists will understand that information and signals maybe represented using any of a variety of different technologies andtechniques. For example, data, instructions, commands, information,signals, bits, symbols, and chips that may be referenced throughout theabove description may be represented by various types of data and/orsignals.

Skilled technologists will further appreciate that the variousillustrative logical blocks, modules, circuits, methods and algorithmsdescribed in connection with the examples disclosed herein may beimplemented as electronic hardware, computer software, or combinationsof both. To clearly illustrate this interchangeability of hardware andsoftware, various illustrative components, blocks, modules, circuits,methods and algorithms have been described above generally in terms oftheir functionality. Whether such functionality is implemented ashardware or software depends upon the particular application and designconstraints imposed on the overall system. Skilled artisans mayimplement the described functionality in varying ways for eachparticular application, but such implementation decisions should not beinterpreted as causing a departure from the scope of the presentinvention.

The various illustrative logical blocks, modules, and circuits describedin connection with the examples disclosed herein may be implemented orperformed with a general purpose processor, a digital signal processor(DSP), an application specific integrated circuit (ASIC), a fieldprogrammable gate array (FPGA) or other programmable logic device,discrete gate or transistor logic, discrete hardware components, or anycombination thereof designed to perform the functions described herein.A general-purpose processor may be a microprocessor, but in thealternative, the processor may be any conventional processor,controller, microcontroller, or state machine. A processor may also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration.

The methods or algorithms described in connection with the examplesdisclosed herein may be embodied directly in hardware, in a softwaremodule executed by a processor, or in a combination of the two. Asoftware module may reside in RAM memory, flash memory, ROM memory,EPROM memory, EEPROM memory, registers, hard disk, a removable disk, aCD-ROM, or any other suitable form of data storage medium now known ormade available in the future. A storage medium may be connected to theprocessor such that the processor can read information from, and writeinformation to, the storage medium. In the alternative, the storagemedium may be integral to the processor. The processor and the storagemedium may reside in an ASIC.

Depending on the embodiment, certain acts, events, or functions of anyof the methods described herein can be performed in a differentsequence, can be added, merged, or left out altogether (e.g., not alldescribed acts or events are necessary for the practice of the method).Moreover, in certain embodiments, acts or events can be performedconcurrently, rather than sequentially.

The previous description of the disclosed examples is provided to enableany person skilled in the art to make or use the present invention.Various modifications to these examples will be readily apparent tothose skilled in the art, and the generic principles defined herein maybe applied to other examples without departing from the spirit or scopeof the invention. As will be recognized, certain embodiments of theinventions described herein can be embodied within a form that does notprovide all of the features and benefits set forth herein, as somefeatures can be used or practiced separately from others. The scope ofcertain inventions disclosed herein is indicated by the appended claimsrather than by the foregoing description. All changes which come withinthe meaning and range of equivalency of the claims are to be embracedwithin their scope. Thus, the present invention is not intended to belimited to the examples shown herein but is to be accorded the widestscope consistent with the principles and novel features disclosedherein.

For purposes of summarizing the invention and the advantages achievedover the prior art, certain objects and advantages of the invention havebeen described herein above. Of course, it is to be understood that notnecessarily all such objects or advantages may be achieved in accordancewith any particular embodiment of the invention. Thus, for example,those skilled in the art will recognize that the invention may beembodied or carried out in a manner that achieves or optimizes oneadvantage or group of advantages as taught or suggested herein withoutnecessarily achieving other objects or advantages as may be taught orsuggested herein.

All of these embodiments are intended to be within the scope of theinvention herein disclosed. These and other embodiments will becomereadily apparent to those skilled in the art from the detaileddescription of the preferred embodiments having reference to theattached figures, the invention not being limited to any particularpreferred embodiment(s) disclosed.

What is claimed is:
 1. A system for processing human related data tomake personalized and context aware decisions with distributed machinelearning at an edge and a cloud, the system comprising: a plurality ofedge computing devices configured to communicate data with each other,the plurality of edge computing devices physically spaced apart fromeach other; at least three sensory devices comprising first, second andthird sensory devices configured to sense a patient's physiologicalsignal in real time to generate a first sensed signal, a second sensedsignal and a third sensed signal and communicate the first, second andthird sensed signals to an edge computing device nearest available tothe first, second and third sensory devices among the plurality of edgecomputing devices; and a core cloud network configured to communicatewith the edge computing devices or the at least three sensory devices,the nearest available edge computing device being in data communicationwith the core cloud network and configured to: receive the first, secondand third sensed signals from the first, second and third sensorydevices; determine when the first sensed signal exceeds a firstthreshold for a first predetermined time; determine when the secondsensed signal exceeds a second threshold for a second predeterminedtime; determine when the third sensed signal exceeds a third thresholdfor a third predetermined time; correlate the first sensed signal andthe second sensed signal to generate a first correlation pattern;determine a lag time between the first sensed signal exceeding the firstthreshold and the second sensed signal exceeding the second threshold;provide the first correlation pattern and the lag time as inputs to afirst long short term memory (LSTM) neural network; correlate the secondsensed signal and the third sensed signal to generate a secondcorrelation pattern; provide the second correlation pattern to a secondLSTM neural network as an input; control the first LSTM neural networkand the second LSTM neural network to provide outputs; and map thepatient to a stage of a medical condition based at least in part on thefirst correlation pattern, the lag time and the second correlationpattern.
 2. The system of claim 1, wherein the nearest available edgecomputing device is further configured to: correlate the first sensedsignal and the third sensed signal to generate a third correlationpattern; provide the third correlation pattern to a third LSTM neuralnetwork as an input; collect a history of states from each of the first,second and third LSTM neural networks; analyze the history of the statesusing an attention network such that an output of the attention networklearns interactions across time and across signals; and summarize ahistory of the interactions using a multi-signal memory aggregator suchthat an output of the multi-signal memory aggregator is fed into adecision making module to map the patient to the stage of the medicalcondition based on the summarized history of the interactions.
 3. Thesystem of claim 2, wherein the nearest available edge computing devicecomprises: a first feature extractor configured to determine when thefirst sensed signal exceeds the first threshold for the firstpredetermined time; a second feature extractor configured to determinewhen the second sensed signal exceeds the second threshold for thesecond predetermined time; a third feature extractor configured todetermine when the third sensed signal exceeds the third threshold forthe third predetermined time; and a first correlator configured tocorrelate the first sensed signal and the second sensed signal togenerate the first correlation pattern and determine the lag timebetween the first sensed signal exceeding the first threshold and thesecond sensed signal exceeding the second threshold as inputs to a firstcell in the first LSTM neural network in an LSTM bank, wherein the thirdfeature extractor is configured to directly feed the third sensed signalto a first cell in the second LSTM neural network in the LSTM bank,wherein the first correlator is further configured to generateadditional first correlation patterns over time into additionaldifferent cells of the first LSTM neural network, wherein the thirdfeature extractor is further configured to provide additional instancesover time when the third sensed signal exceeds the third threshold asinput signals into additional different cells of the second LSTM neuralnetwork, wherein the cells of each of the first LSTM neural network andthe cells of the second LSTM neural network in the LSTM bank areconfigured to be fed into a fully connected neural network to generateattention map coefficients that are component-wise multiplied with thecells of the first LSTM neural network and the cells of the second LSTMneural network to generate an attention map, wherein the attention mapis configured to be fed into the multi-signal memory aggregator toaggregate multiple signal memories over time to prepare an optimal inputinto the decision making module, and wherein the decision making moduleis configured to make a decision to map the patient to the stage of themedical condition based on the optimal input received from themulti-signal memory aggregator.
 4. The system of claim 1, wherein thefirst sensed signal, the second sensed signal and the third sensedsignal are of different modalities, wherein the first correlationpattern, the second correlation pattern, a state of the first LSTMneural network and a state of the second LSTM neural network areconfigured to be fed into a first multi-modal LSTM neural network,wherein the second correlation pattern, the third correlation pattern,the state of the second LSTM neural network and a state of the thirdLSTM neural network are configured to be fed into a second multi-modalLSTM neural network, and wherein the states of the first, second andthird LSTM neural networks, and outputs of the first multi-modal LSTMneural network and the second multi-modal LSTM neural network areconfigured to be fed into a multi-signal memory aggregator.
 5. Thesystem of claim 4, further comprising an attention function processorconfigured to: receive one of the first, second and third sensed signalsas an input signal; find one or more certain patterns of the inputsignal; and categorize the input signal and generate the attention mapcorresponding to the certain patterns before being correlated.
 6. Thesystem of claim 4, further comprising an attention function processorconfigured to: receive one of the first, second and third sensed signalsas an input signal; find one or more certain patterns of the inputsignal; and categorize the input signal and generate the attention mapcorresponding to the certain patterns before being an input of one ofthe first or second multi-modal LSTM neural networks.
 7. The system ofclaim 3, wherein the nearest available edge computing device isconfigured to present the attention map to a healthcare professional asdocumentation to support the determination of the stage of the patient'smedical condition.
 8. The system of claim 2, wherein the decision makingmodule comprises at least one fully connected neural network.
 9. Thesystem of claim 8, wherein the decision making module is configured togenerate a scalar quantified risk score.
 10. The system of claim 8,wherein the fully connected neural network comprises a scaled sigmoidactivation function.
 11. The system of claim 8, wherein the decisionmaking module comprises an argmax function configured to operate on anoutput of the fully connected neural network.
 12. The system of claim11, wherein the decision making module is configured to generate abinary format prediction.
 13. The system of claim 11, wherein the fullyconnected neural network comprises a unit for each class of amultiple-class classification and wherein the output of the argmaxfunction is a probability of the input data belonging to each class ofthe multiple-class classification.
 14. The system of claim 2, whereinthe nearest available edge computing device comprises at least one ofthe first to third LSTM neural networks and the decision making module.15. The system of claim 1, wherein the nearest available edge computingdevice is configured to buffer and align at least one of the first,second and third sensed signals before being correlated.
 16. An edgecomputing device for processing human related data to make personalizedand context aware decisions with distributed machine learning at an edgeand a cloud, the edge computing device comprising: a memory storingcomputer executable instructions; and a processor in data communicationwith the memory and, when executed by the executable instructions,configured to: receive a first sensed signal, a second sensed signal anda third sensed signal obtained in real time from sensing a patient'sphysiological signal from first, second and third sensory devices,determine when the first sensed signal exceeds a first threshold for afirst predetermined time, determine when the second sensed signalexceeds a second threshold for a second predetermined time, determinewhen the third sensed signal exceeds a third threshold for a thirdpredetermined time, correlate the first sensed signal and the secondsensed signal to generate a first correlation pattern, determine a lagtime between the first sensed signal exceeding the first threshold andthe second sensed signal exceeding the second threshold, and correlatethe second sensed signal and the third sensed signal to generate asecond correlation pattern; a first long short term memory (LSTM) neuralnetwork configured to receive the first correlation pattern and the lagtime from the processor; and a second long short term memory (LSTM)neural network configured to receive the second correlation pattern fromthe processor, wherein the processor is further configured to: controlthe first LSTM neural network and the second LSTM neural network toprovide outputs; and map the patient to a stage of a medical conditionbased at least on the first correlation pattern, the lag time and thesecond correlation pattern.
 17. The edge computing device of claim 16,wherein the processor is further configured to: correlate the firstsensed signal and the third sensed signal to generate a thirdcorrelation pattern; and provide the third correlation pattern to athird LSTM neural network as an input, wherein the processor isconfigured to make a decision on outputs of the first, second and thirdLSTM neural networks.
 18. The edge computing device of claim 17, whereinthe processor is further configured to make the decision by performing ascattering function on the outputs of the first, second and third LSTMneural networks.
 19. The edge computing device of claim 16, wherein theprocessor is further configured to: correlate the first sensed signaland the third sensed signal to generate a third correlation pattern;provide the third correlation pattern to a third LSTM neural network asan input; collect a history of states from each of the first, second andthird LSTM neural networks; analyze the history of the states using anattention network such that an output of the attention network learnsinteractions across time and across signals; summarize a history of theinteractions using a multi-signal memory aggregator; and feed an outputof the multi-signal memory aggregator into a decision making module tomap the patient to the stage of the medical condition based on thesummarized history of the interactions.
 20. The edge computing device ofclaim 16, wherein the processor comprises: a first feature extractorconfigured to determine when the first sensed signal exceeds the firstthreshold for the first predetermined time; a second feature extractorconfigured to determine when the second sensed signal exceeds the secondthreshold for the second predetermined time; a third feature extractorconfigured to determine when the third sensed signal exceeds a thirdthreshold for a third predetermined time; and a first correlatorconfigured to correlate the first sensed signal and the second sensedsignal to generate the first correlation pattern and determine the lagtime between the first sensed signal exceeding the first threshold andthe second sensed signal exceeding the second threshold as inputs to afirst cell in the first LSTM neural network in an LSTM bank, wherein thethird feature extractor is configured to directly feed the third sensedsignal to a first cell in the second LSTM neural network in the LSTMbank, wherein the first correlator is further configured to generateadditional first correlation patterns over time into additionaldifferent cells of the first LSTM neural network, wherein the thirdfeature extractor is further configured to provide additional instancesover time when the third sensed signal exceeds the third threshold asinput signals into additional different cells of the second LSTM neuralnetwork, wherein the cells of each of the first LSTM neural network andthe cells of the second LSTM neural network in the LSTM bank areconfigured to be fed into a fully connected neural network to generateattention map coefficients that are component-wise multiplied with thecells of the first LSTM neural network and the cells of the second LSTMneural network to generate an attention map, wherein the attention mapis configured to be fed into a multi-signal memory aggregator toaggregate multiple signal memories over time to prepare an optimal inputinto a decision making module, and wherein the decision making module isconfigured to make a decision to map the patient to the stage of themedical condition based on the optimal input received from themulti-signal memory aggregator.
 21. A method of processing human relateddata to make personalized and context aware decisions with distributedmachine learning at an edge computing device in communication with acloud, the method comprising: receiving, at a processor of the edgecomputing device, a first sensed signal, a second sensed signal and athird sensed signal obtained from sensing a patient's physiologicalsignal from first, second and third sensory devices; determining, at theprocessor, when the first sensed signal exceeds a first threshold for afirst predetermined time, determining, at the processor, when the secondsensed signal exceeds a second threshold for a second predeterminedtime, determining, at the processor, when the third sensed signalexceeds a third threshold for a third predetermined time; correlating,at the processor, the first sensed signal and the second sensed signalto generate a first correlation pattern, determining, at the processor,a lag time between the first sensed signal exceeding the first thresholdand the second sensed signal exceeding the second threshold;correlating, at the processor, the second sensed signal and the thirdsensed signal to generate a second correlation pattern; receiving, at afirst long short term memory (LSTM) neural network of the edge computingdevice, the first correlation pattern and the lag time; receiving, at asecond LSTM neural network of the edge computing device, the secondcorrelation pattern; controlling, at the processor, the first LSTMneural network and the second LSTM neural network to provide outputs;and mapping, at the processor, the patient to a stage of a medicalcondition based at least on the first correlation pattern, the lag timeand the second correlation pattern.
 22. The method of claim 21, furthercomprising: correlating, at the processor, the first sensed signal andthe third sensed signal to generate a third correlation pattern;receiving, at a third LSTM neural network of the edge computing device,the third correlation pattern; collecting, by the processor, a historyof states from each of the first, second and third LSTM neural networks;analyzing, at an attention network of the edge computing device, thehistory of the states to learn interactions across time and acrosssignals; summarizing, at a multi-signal memory aggregator of the edgecomputing device, a history of the interactions; feeding, by theprocessor, an output of the multi-signal memory aggregator into adecision making module of the edge computing device; and mapping, at thedecision making module, the patient to the stage of the medicalcondition based on the summarized history of the interactions.
 23. Themethod of claim 21, further comprising: receiving first correlationpattern inputs at a first cell of the first LSTM neural network in anLSTM bank; directly receiving the third sensed signal when the thirdsensed signal exceeds the third threshold at a first cell in the secondLSTM neural network in the LSTM bank; receiving additional firstcorrelation patterns over time into additional different cells of thefirst LSTM neural network; receiving, by the processor, additionalinstances over time when the third sensed signal exceeds the thirdthreshold as input signals into additional different cells of the secondLSTM neural network; generating, by an attention network of the edgecomputing device, attention map coefficients based on the cells of eachof the first LSTM neural network and the cells of the second LSTM neuralnetwork in the LSTM bank to be fed into a fully connected neuralnetwork; generating, by the attention network, an attention map based onthe attention map coefficients that are component-wise multiplied withthe cells of the first LSTM neural network and the cells of the secondLSTM neural network; feeding, by the processor, the attention map into amulti-signal memory aggregator that is configured to aggregate multiplesignal memories over time; and mapping, at a decision making module, thepatient to the stage of the medical condition based on the aggregatedmultiple signal memories received from the multi-signal memoryaggregator.
 24. The method of claim 21, wherein the edge computingdevice is a nearest available edge computing device to the patient amonga plurality of edge computing devices comprising the first to third edgecomputing devices being in data communication with the cloud.
 25. Themethod of claim 21, further comprising: receiving a request for servicefrom a sensory device of the patient; locating the patient's sensorydevice; determining that the edge computing device is a nearestavailable edge computing device to the patient sensory device of aplurality of edge computing devices comprising the first to third edgecomputing devices being in communication with the cloud; and assigning aservice slot to the patient's sensory device.
 26. The method of claim25, wherein determining the nearest available edge computing devicecomprises: receiving, at the plurality of edge computing devices, asignal sent by the patient's sensory device; measuring strengths of thesignal received by the plurality of edge computing devices; comparingthe strengths of the received signal; and determining an edge computingdevice to have a strongest signal strength as the nearest available edgecomputing device.
 27. The method of claim 21, further comprising:buffering and aligning one of the first sensed signal or the secondsensed signal before the correlating.
 28. The method of claim 21,further comprising: finding one or more certain patterns of the firstsensed signal or the second sensed signal; and generating an attentionmap corresponding to the certain patterns before the correlating. 29.The method of claim 28, further comprising: presenting the attention mapto a healthcare professional as documentation to support thedetermination of the stage of the patient's medical condition.
 30. Asystem for processing human related data to make personalized andcontext aware decisions with distributed machine learning at an edge anda cloud, the system comprising: a plurality of edge computing devicesconfigured to communicate data with each other, the plurality of edgecomputing devices physically spaced apart from each other; at least twosensory devices comprising first and second sensory devices configuredto sense a patient's physiological signal in real time to generate afirst sensed signal and a second sensed signal and communicate the firstand second sensed signals to a first edge computing device nearestavailable to the first and second sensory devices among the plurality ofedge computing devices; and a core cloud network configured tocommunicate with the edge computing devices or the at least two sensorydevices, the nearest edge computing device in data communication withthe core cloud network and configured to: receive the first and secondsensed signals from the first and second sensory devices; determine whenthe first signal exceeds a first threshold for a first predeterminedtime and subsequently determine when the second signal exceeds a secondthreshold for a second predetermined time; correlate the first signaland the second signal to generate a first correlation pattern; determinea lag time between the first signal exceeding the first threshold andthe second signal exceeding the second threshold; and provide the firstcorrelation pattern and the lag time as inputs to at least two recurrentneural networks (RNNs) operatively connected to each other to provide aninput to a decision making module to map the patient to a stage of amedical condition based at least on one or more of the first correlationpatterns and the lag time.